Too many variables involved that work against this idea
(mainly the fact that you are depending on individuals that have no business/employment interest with the projects. Thus the likelihood that the results might be skewed is very high)

however if you build an application that lets the end user download it and simply use it to run the check against

---> doing such you can compare multiple source feedback to assess how competent the tests were. One person could be wrong, but 100's of them testing the same product across the globe cant be wrong about the same thing at the same time.

Hey Gen

Thanks for the comment.

I agree with what you are saying.

One of the difficult parts of new mobile applications is that they often require network compatability. There are filters and specific ports from carriers which sometimes do not allow network access to servers on the web.

The full end functionality of the application is not really the test. (hopefully that has been tested already)

Does the application fit within the screen?

Can the enduser access the network part of the application?

If this is a location based application, do the maps show properly and is the end location correct or within predifined ranges.

Is the memory in the handset adequate?

Is the application quick enough on the specific carrier or is it painfully slow? Is it impossible to use at rush hour at the end location?

These are the types of things that drive application developers crazy.

I look forward to more comments or questions from you generic_idea_machine!

well imagine a "proxy scape"

An application that can you can "rent" out to the big guys (cariers) to use.

- So the carriers get to tweak and administer the back end of the application, specifying ports, mode of transfer, technology (gprs, edge, cdma, umts)

- and then you have the populace flock to the site and run tests as such. Based on the handset, the application redirects the user to the requisite test site.
- Based on the assessment, you have another application running in the back that would analyze the performance metrics. And provide the feedback back to the carrier.

- Carrier saves money by outsourcing testing.
- Beta testers make some dough in the spare time

Hey Gen

I hear what you are saying.

Application developers have emulators to test up to a certain point the application.

However in many cases it is the developer who has to support the end user application. There is no point in a finger pointing game to determine the cause for a problem on a end user problem.

It is the application developer ( or publisher) who has their reputation on the line. They are the one who need the application to work on a specific carrier with a specific handset.

A specific handset for one carrier may be completely different in OS than the same handset with another carrier.

The carriers do test up to a point but in many cases the applications are commodities to the carriers.

The statistical results of your "proxy scape" will give some parameters but in my personal experience, unless you test the specific application on a carrier handset, you cannot be sure it will work.

As well, carriers do not always give out test phones. It is up to the application developer to buy the phone. Try buying a phone in a country with a carrier without a home address in that country. Even as a developer it is virtually impossible.

I like the slogan "certified crowd tested", I think it would sell...

neat idea. Can even ask the "testers" to provide feedback on how to improve the application. Mind you, I see the "testers" as "users" much more than "testers". May be the crowd can even be pay extra bonus for catching bugs. And extra $ for meaningful features that the company is interested in developing.

Don't really like "Test 'er Test 'er good....". "certified crowd tested" is slightly better but there is probably a better one still.

Hey Kempton

thanks for the feedback....

Yes I believe application developers would love to get feedback in regards to GUI and other things.

It is way easier to make changes before a release goes out then when it is released.

There's definitley a lot of unique challenges when it comes to testing mobile apps and a solution that can address these will be a winner I think.

thanks drh

The challenges are very unique and expensive. The mobile application market is getting bigger and bigger.

According to analysts Juniper Research, the total global mobile entertainment market (including gambling, adult content, games, music and TV), is currently worth over $17bn.

The firm forecasts this will grow to $47bn by 2009 and $77bn by 2011, as broadcast mobile TV and mass market casual games really take off.

This is does not include business and other applications.

Taking it one step further .. how about a Mobile Developers Union? The union would be able to negotiate with both the carriers and equipment manufacturers on behalf of developers that are members of the union. Ok, crazy idea .. I know.

But back to this idea, I didn't really understand the proposed business model .. and I think I have a better one. The company harnesses a network of 100,000 mobile phone testers from all over the world, with every combination of carrier and phone. They offer several testing plans. For open source apps, they will run limited tests free of charge. For small businesses or individual developers, they offer cheap one-time or repeated testing plans, ranging in the hundreds of dollars. Or instead, this type of customer could join as a tester and get free testing in return. For big companies, the testing fees range in the tens of thousands of dollars.

The individual testers are not paid in cash, but instead they get "perks". The main perk will the ability to leverage the network for testing their own apps free of charge, but also to developer forums, online tools, early access to pre-released software, etc. When getting free testing
in return for doing testing, the system would ensure that members
do about 3x as much testing than they receive, in order to create
a profitable model.

Another arm of the business model would be advertising revenues to the developer and testers network.

Hey Branchcut

Branchcut Thanks for the feedback!

I like your business models and perhaps they would work.

I was looking at more of a bidding system where application developers and end testers would negotiate online to get testing done for a specific application.

For example, application developer abc says they will pay $50 to do a test on a specific model of phone on a specific carrier.

Tester xyz says they will do it for $75 dollars. abc and xyz negotiate until a price is reached. The money stays in escrow until the testing is done. On successful test, the money is released. The application developer pays a percentage fee to cell_tester (ie 15%) and the test is complete.

The more complex of a test, the more the tester will want.

The test procedures will have to be complete and upfront.

I have heard from carriers and application developers re this service and they both have agreed that there is a very definite need for this. Carrier testing labs cannot even get enough of the handsets to test the products.

In North America, there are very few open source mobile applications due to the nature of the garden wall mentality of the carriers.

I think a bidding system will ensure the right price for both the application developers and the testers. The market will even out for the appropriate application being tested and the complexity of the test required.

I think a paypal cash system will work out better than perks. The best testers will be rewarded because of the ranking system.

I am currently developing a mobile application and have had feedback from three testers. All three have different screen resolutions! I think this idea is excellent. Don't beta testers usually do it for free though?

Good idea, the benefits are obvious and potential for profit are clear.

My main concern like Generic mentioned early on would be guaranteeing the quality of service, due to human nature and if the tests will actually be carried out properly. The flip side of this is also if you’re an application developer who’s hired 20+ people globally are you going to bother rating your testers accurately and fairly?

I’m thinking that perhaps the best way to operate would be to provide the framework for testers to operate their own groups. Testers would set up groups of their known associates around the globe and bid for work as a unit and then be rated as a unit. A group of known associates would work better then many individuals as they will filter out the dross, as ultimately their rating requires everyone to work together.

Just a thought anyway.

I like the idea and think it has some good potential. In a magazine article Symbian claims that the USA is reaching the golden # of cell phone users, roughly 75%. In Japan and likely Europe, that 75% figure was a time when wireless carriers started needing application developers, not the other way around as it had been in the past. Those application developers are going to need this type of service. Down the road, if "Certified Crowd Tested" gains enough recognition the wireless carriers will look for the depth that this service could offer. BTW, the magazine puts the wireless data market at 16 billion - and this service could, in a sense, be a backbone for that industry.

I am currently developing a mobile application and have had feedback from three testers. All three have different screen resolutions! I think this idea is excellent. Don't beta testers usually do it for free though?

If the application developer can get the test done for free, (unlikely but possible) the service will still charge a minimum fee to the application developer. You have to remember that application developers spend thousands of dollars configuring various handsets for testing.

My main concern like Generic mentioned early on would be guaranteeing the quality of service, due to human nature and if the tests will actually be carried out properly.

I agree. Quality would be required. The groundwork for testing etc would be key.

The possibility using groups to test would also be possible..Ie a Sprint or Nextel group could test in those markets.

you folks have "thang" going on - back and forth, like that. it's great to see idea malignancy...in social theory, this is called particpant driven research - it is something that we participate in every day and it needs to become a more notable part of any business paradigm.

Hey wicker

Thanks for your comments...

Idea malignancy is an interesting phrase. What do you mean by this?

Certainly, from the latin malignant literally means "born to be bad" and is obvious for its cancerous meaning.

I look forward to your reply.

Much needed.

Example. I have on my P990i three WAP push messages - unsolicited advertising that when opened take you to a wap site. If I scroll them into view on my phone the message list crashes, with the error "not found"

I can't delete them, nor can I delete the 15 messages either side incase the rogue messages come on screen.

The handset manufacture, network operator and the message sender all have no idea why the messages are a problem or how to delete them - without deleting all my messages.

Better testing / standards adherance would benefit the whole mobile industry.

Hey Allan

Thanks for your comments.

I totally agree.

This idea would likely appeal to the youth market. Perfect for downloading and testing free games. Give them incentive to be a "tester" by offering free minutes (if appropriate) or points to get free stuff which would be provided by whoever wants their mobile product tested. Kind of like a "Nielsen Household", it could become a "privilege" to be a tester.

Thanks for the comments smoir1.

You are absolutely correct that this could be used as a marketing tool as well for the application developer.

Market and functionality tests would be great for feedback.

I attended a lecture from Steven Cheng of Innaworks who deals with this problem and many other issues surrounding this market. Steven ports tests and optimizes for essentially every major software companies in the mobile arena. Some of the insight I gained:

Its a hard slog and the carriers are god, they control everything with a horrific manner.
This is a highly competitive market. Often they will make versions of a cellphone app for every popular model.
Sending it to a porting house in India is cheep.
Relying on users to provide the right feedback would be tough.
Unlike computer software betas, buggy cellphone software is not fun!

Hey Patmania

Thank you for your comments.

I will try to answer your concerns.

"Its a hard slog and the carriers are god, they control everything with a horrific manner."

I know and sometimes the carriers do not test adequately. The testing labs sometimes do not have all the handsets to test. (I have had emails from mobile operators who would love this service because the handset costs come out of their departmental budget)

"This is a highly competitive market. Often they will make versions of a cellphone app for every popular model." True but we are not talking about the cost of porting (ie programming for various models) but the cost of testing can be huge especially with networked and location applications.

"Sending it to a porting house in India is cheep" True and that is why testing costs so much. You cannot test a USA Sprint handset application in India because Sprint does not operate in India. (ie the handset application may require specific networking parameters and handset parameters which are common to the USA operations only) A networked application cannot be tested in India because the network will be different.

"Unlike computer software betas, buggy cellphone software is not fun!" That is why the testers should be paid by the auction system. I anticipate testers getting paid to do their job. If a particular test was really tough, the tester will bid higher to do the work, the next time. Let the market decide what the costs will be.

I hope this answers your concerns.

Take a look at the business at http://www.cambrianh...ss/view/cell_tester/

quick questions:
-will you support Java as well as BREW?
- since you mentioned it's an auction, I assume testers will bid to do
a testing job, and cell companies will accept the lowest bids? A model
which increases revenue for testers would be beneficial to them
- what constitutes testing? what are the methods, metrics etc
- how will vet testers' experience, abilities and qualifications?
- how will you verify that the work was done, before payment?

Nifty idea fish! Good luck with it!

Answers to TheGuru...thanks for those.

quick questions:
-will you support Java as well as BREW?

Whatever the endusers phone will support will be tested. If it is brew or j2me or symbian

- since you mentioned it's an auction, I assume testers will bid to do
a testing job, and cell companies will accept the lowest bids? A model
which increases revenue for testers would be beneficial to them

the bidding will go both ways. Let the market decide the price to do the job.

- what constitutes testing? what are the methods, metrics etc

The methods etc. will be the application developers responsibility. The testers will see the methods etc and bid what they believe to be the appropriate cost to do so.

- how will vet testers' experience, abilities and qualifications?

The testers will be ranked by the application developers and vice versa. The testers will show what experience they have online.

- how will you verify that the work was done, before payment?

The application developers will determine the work to be done. If it is a network application they will see the traffic. The money will be kept in escrow until both side are happy with the work.

I hope this answers your questions.

Great Idea!

Keep going!

They all ready do this thing called beta product testing. So, there is no way any one would buy this "service" that they are all ready are doing!

The problem with beta testing or any testing is that the application developers cannot get the phones to test without paying huge amounts of cash. (if they can get the phones at all)

If they are buying the phones with the plan is is because they need a technical to test not the average home user. Usually they get review models from the companies at no cost.

For beta testing usually they pay groups of people next to none to get people to try a program for free. Then they will usually send them a free copy of the program after they release the full copy.

So, I don't see why they would want to use this service when they all ready do it themselves.

like it!

Nope

I'm in development of mobile games. I can tell you now it doesn't work at all like you think... Companies want a specific game on specific phones and that's it.

The games give user a reason to buy a specific phone and at the same time, if you want to play the new version of a game or app.... You have to buy a new phone.

Your idea will never work until companies are ready to change their ways... with I don't.

Sorry about this.

]V[oogy

robertman & Kevin_Cox: I think you are missing the point here.

Sure, if you're Microsoft you can get beta testers easily but for those of us with limited resources this service will be a godsend.

The blastfm alpha is currently only confirmed as working on two different devices, both of them S60 3rd edition. There are almost as many different j2me implementations as there are devices and they all differ in subtle ways particularly regarding the mmapi.

I will be happy to pay for this service when it launches and there are many other small developers who I believe will pay also.

Hey Moogy

Thanks for your comments.

I have to disagree with you. I am the former president of Blister Entertainment (first mobile location based games in North America)

The difficulty of testing games on phones is difficult because it is hard to get the phones on a specific network.

I have had responses from the testing departments at mobile companies who applaud this because they cannot get the phones themselves.

This is especially true on networked games and applications.

Remember this is a cost and leverage business. The average cost of testing on a specific phone

Remember this idea will reduce the cost of testing.

"The games give user a reason to buy a specific phone and at the same time, if you want to play the new version of a game or app.... You have to buy a new phone."

Yes this is true. However if you wish to test on a global basis, you need to test in each market on different phones with different networks.

Very expensive.

Why not use the handsets that people already have and have them test. You do not have to fly teams of people out to test in a market. Remember,

Much cheaper.

The slogan says it all..."CERTIFIED CROWD TESTED"

BTW I have downloaded multiple mobile games on my handset, that were buggy and in some cases did not work at all.

I don't have a good knowledge about mobile applications but this idea seems to be a great tool to resolve some incompatible issues… It has my vote!!

fish, I'll give you a five (since I usually only give a 1 or 2) since your idea is:
1) actually viable 2) there is a need 3) people are willing to pay. to top it off you answered all of everybody's questions intelligently, showing you'd thought this through. This is by far the best idea I've seen, and is leaps above most of the crud that is on here. I hope this idea makes it on it's own merit, and competes with the technically flawed and BS ideas that have floated to the top. Don't forget - the cream is not the only part that rises to the top! :(

Hey TheGuru

Thank you for your kind comments.

I do disagree with you on 1 point however. I do believe there are good ideas out there that are possible and feasible. The difficulty with some of the ideas is the communication of what they wish to do.

I wish all the ideas a good chance and wish them all good luck.

Great idea and unlike most submitted on CH, could actually be done without a multi-million dollar budget and a few spare years of product development time.

If you need a hand in Australia, let me know. I am guilty of always buying the latest and greatest nerdy smart phones thanks to my passion for LBS.

me gusta

Sounds like a good idea. I can't believe there is a gap like you describe, it's almost bizarre. So many users would be willing testers, it's logical to use them for it.

What I don't understand is why anyone would (previously) have flown testers to various locations to do the testing on local technology.

Why can't a product developer simply place an ad on whatever medium will reach the intended market, and get testers to sign up remotely?

If all it takes is a phone with a local account, why was it necessary to fly people anywhere?

There is something missing in my understanding of the scenario, but as I understand it the solution is simple and sensible.

I guess if you are creating this network of testers as a stand-alone business, then you are not linking it to just one particular product. The tester network could be used for all kinds of market and beta testing, if it became large enough.

I have never heard of any companies flying people out to a location just to beta test some software. My guess is that there were other reasons "mobile location based games".

Software companies that I know for mobile simply use the review units by the phone distributor or have what is know as a beta test. Basically like I said before. Usually they pay groups of people next to none to get people to try a program for free. Then they will usually send them a free copy of the program after they release the full copy.

A lot of the of the time they don't even pay them, just offer them something as simple as some pizza and drinks.

I think most of us are familiar with the concept of beta testing. If you are huge already or are willing to do a huge marketing pre-launch effort a la Joost then beta testers are easy to find. But, with an new product it is not so easy to get it out there.

Hi Jill

Thank you for your comments.

"Sounds like a good idea. I can't believe there is a gap like you describe, it's almost bizarre. So many users would be willing testers, it's logical to use them for it."

The mobile space is a strange place. It is not easy to find testers in various markets. Remember this is a global market with various carriers. Each carrier is different. Every phone is different.

"What I don't understand is why anyone would (previously) have flown testers to various locations to do the testing on local technology."

Imagine you are an networked application developer in Toronto. If you wish to test an application for Sprint in the USA, you cannot test the application in Toronto because your phone would not be on the Sprint network. You would be testing likely through Bell Mobility or Telus in Toronto. Each network is different.

"Why can't a product developer simply place an ad on whatever medium will reach the intended market, and get testers to sign up remotely?"

Well the difficulty would be, where would you place an ad? What are the timing issues with this? This idea would essentially do this.

"If all it takes is a phone with a local account, why was it necessary to fly people anywhere?"

Here again if you are an small application developer in Toronto, try to get a phone on the Sprint network in the USA without a US address. If you cannot show an id with an American address, Sprint will not sell you a phone with a plan. As well, if you did have an address, the phone may cost up to $1000 with a plan. Then you need people to go the US to test the application. The test may only be an hour long, but it has cost thousands of dollars to test the application.

"I guess if you are creating this network of testers as a stand-alone business, then you are not linking it to just one particular product. The tester network could be used for all kinds of market and beta testing, if it became large enough. "

I think you are right on here. Imagine a large test of a game on various phones and getting feedback technically but also on gameplay. (each phone has a different speed of CPU and memory usage)

thanks again Jill.

Hey Siddey

As soon as we are up, we will get you up in Australia!

...Make it so!
If you need anything, let me know.

honourable mention "best painkiller" June 21 2007 round of voting see

http://www.cambrianh...pen-render-networks/

Work is being completed to have an alpha site of Mob4Hire ready for mid September 2007.

The site will have a contest for signup which will be a new iPhone or $500US. New developers will be given a $200 signing bonus which can be used for their first project.

BTW all the work completed will be done by CH members. Anathema is cranking on the code and we will use CH member web developers to get an appropriate style. Fossiloflife created the logo through the bazaar.

I continue to get responses from people in the mobile industry who understand the importance of this project.

Annievee (ch member) wrote this pm to me. "There are a couple of alternatives out there for people to test (Mobile Complete, SysOpen Digia etc..) but I like the fact that this brings in people who are not as close to the application to test...kind of like beta testing network vs. a remote controlled access situation. "

If you have questions, please post.

Cheers Paul Poutanen aka fish99

This is a great idea! I think not only does it capture the testing functionality aspect but you also get a number great business benefits.

Your application will then be tested by potential customers. Great way to plant the seed and feed the buzz.

The NUMBER 1 REASON MOBILE APPLICATIONS FAIL is because of poor usability. It would be great to get people who are so attached to the unique features of their own handheld test this so that they can tell you if something is natural, unnatural, cumbersome or in line with device based shortcuts, navigation and consistency with other apps. These constituents are the next best thing to usability specialists if you can't afford them ( Although I highly recommend them)

For all of the consumer applications out there, this is a perfect way to test.

The next step is to figure out how to leverage this on an enterprise/business scale where you may not own the backend and it may reside inside a firewall somewhere not accessible to the public.

It's similar to what Centercode and other testing groups do, so I don't see what it would't work.

Having one that just focuses on mobile could be useful.

Sounds like this fills a real and growing need and the opening is created by the still-new concept of crowdsourcing. Crowd feedback is perfect because the market is exactly the crowd.

Looking at it from the purchaser's point of view, if you had trained and certified testers, that would help. I would have more confidence that the people I engaged would know what they were doing, could follow directions, would follow through, etc.

Training and testing (along with registration of the user demographics, phone and service type) could be provided online and for free, with free goodies (games, songs, whatever) for signing up and for referring others. This is how you can get your network up worldwide with little cost.

Developing a protocol for testing would also be useful, so the testers would have some idea of what to expect and could be trained in it. If there are industry standards already, this may not be necessary, just train the testers in this standard.

Such training and certification (not just ratings after the fact, but including those) allow you to provide more assurance of useful results to the purchaser of this service. That would aid sales and adoption.

Relying solely on the escrow method could be tedious, time-consuming and possibly litigious if someone really didn't perform but insisted they did. Also, a developer could screw the testers if they are so inclined by simply not agreeing they did the job well enough. A simple ambiguity, a culturally based misunderstanding, or poorly written instructions could lead to this as well. How will those be handled? Will you have a pre-flight phase of testing the test (possible premium service)? Saying things unambiguously is harder than it seems, particularly for experts who are too close to their product.

An arbitration clause in the standard contract might be useful.

Having networks of testers set up might be useful. You mentioned specific brands of phone, but demographic sets, geographic sets, and more-specifically identified sets (so many of such an age range or occupational range in these countries with these phones, etc.) might also be useful. This is also premium service, allowing higher charges, as opposed to just putting it out and seeing which testers respond.

Further simplifying the process for participants on both ends would be massed bidding, not one on one. Identified networks could have a designated bidder to get them jobs, meaning one negotiation and one payment, with professional level relationships and internal discipline for nonperforming testers (more instruction, warnings, expulsion, as appropriate).

A single tester could be in multiple networks, of course (model of handset, service, age, country, occupation, income bracket, etc.)

This process could be automated so when a client identifies the set of testers desired a negotiator is selected for that specific group and handles the process.

Negotiator can be a special role with results and satisfaction tracked. For each test a negotiator can be selected by vote, by the testers. Negotiators get a percentage of the payment to the testers, so their allegiance is clear. It means less time and less work on all sides with fair payment to the testers, as a knowledgeable tester can be reasonable in terms of the market and may even suggest new possibilities to the customer.

This business model seems to work for the games industry very well, so i think it would also be successful in the field of mobile computing.

Hey annievee

I appreciate your kind comments.

"Your application will then be tested by potential customers. Great way to plant the seed and feed the buzz."

Marketing feedback will be a great way to determine if you are going the right direction or if your user interface is too hard to use.

"The next step is to figure out how to leverage this on an enterprise/business scale where you may not own the backend and it may reside inside a firewall somewhere not accessible to the public. "

That is an interesting point and certainly something we will look into in the future. To start though, we will aiming for simple use by both the testers and developers.

Thank you CarlenLea for your comments. We will be only focussing on the niche of mobile testing.

Thank you wizbot for your insights to the issues we are working with.

The key to market penetration will be the cost and ease of use of the site.

"Looking at it from the purchaser's point of view, if you had trained and certified testers, that would help. I would have more confidence that the people I engaged would know what they were doing, could follow directions, would follow through, etc. "

Certification is something we are looking at. There are testers out there who will have the qualifications to be certified. We are looking for both testers and developers to be able to rank each other on the job done.
If the tester did a good job, the developer will rank the tester appropriately. If the developer tester documentation was not very good, the tester will note that and rank the developer as such.

"An arbitration clause in the standard contract might be useful."

We will have an arbitration clause in the contract.

"A single tester could be in multiple networks, of course (model of handset, service, age, country, occupation, income bracket, etc.) "

This very true and we are designing as such.

Thank you again for the time to comment on this!

Thank you zentropy for your comment!

hey fish99,

just voted for your idea because of the update you sent me.

i'm fairly new to CH so seeing that ideas & people behind them
are real gives me confidence to get involved in more good projects.

i hope yours goes all the way for you and your team. it is clear that
you are working diligently on this one.

ciao
voxgirl

Hey voxgirl

Thank you for your feedback.

Even though there are nicknames for the people here, these are real people with real projects.

Find one that tickles your fancy or come up with your own idea.

I wish you luck.

Paul

mob4hire.com = 403 Error

Hi Kevin

Mob4hire will up in alpha in mid September. I will post to all when it is ready.

Cheers

Paul

i dunno..i dont see this as viable enough to be a long term business.

Nice one. Go ahead!

well man u again have ma vote! :) and am waiting for September

Good idea, but hard to realize. Just go for it =D

Nokia give away phones for testers...what make a multiple user click...Multiple incentives?

Sounds like a great tool for developers. Good luck.

from eZeitgeist"i dunno..i dont see this as viable enough to be a long term business."

Thanks for your comment. Do I see this as being a 20 year company with the exact same business model? NO

Do I see interest from the developer networks with the same model in the next 3 years? YES

You have to realize that the mobile application business is very tough and difficult. Can I help the developers (and the carriers btw) in the existing market by giving them a less difficult testing solution. YES

Eventually things will get easier. The difficulty is there are many manufacturers of handsets. In order to make their quarterly results, they have to be very competitive. There are many differences in existing handsets.....size, screen, memory, carrier speed, price and processing speed.

Do you realize when (EA) Jamdat Bowling came out they had over 300 versions of the code! Imagine a small developer trying to get that many handsets to test on. Over half their costs in producing that game was porting and testing!

Firefox, fosiloflife and JelmerBV... Thank you for your comments!

From Rosary_Monday "Nokia give away phones for testers...what make a multiple user click...Multiple incentives? "

Thank you for your comments.

Nokia does give handsets to developers once you are a part of their developers program (which costs money I believe). If there are testers out there on this program, please sign up on Mob4Hire when we launch.

The real difficulty in testing even on these phones is in NA anyway, you have to get subscriptions on the carriers ($500 to $1000 a year) to confirm if a network application aimed at Sprint will work.

Mob4Hire will allow testers to use multiple handsets on multiple carriers.

bcforrester....Thank you for the words of encouragement!

I think a lot of folks who would be interested in this would be geeks and tech-savvy folks, thus hopefully skewing the quality of the results in your favor. Tester rating systems would be good as well, I like this all around. Make part of the reward a free copy of the SW being tested! ;]

I've read most of the comments but not all, so I apolgize if I'm repeating anything. I've been working with a start-up mobile application company, and it's been the cost of testing and certification (required by most carriers) that really seems to be an obstacle in North America. We've worked with http://www.tirawirel...g-certification.asp, who as you see offers the services you are proposing. There are a few other developer tools like those being offered by Bell and Nokia. I believe the good folks at Sun even have a JAVA certification process.

Definitely a growth area. I like the idea of "crowdsourcing" testers, but I see a couple of major obstacles. The first being acceptance of certification from the carriers. They second is the being able to ensure that a all phones required are tested reliably. I'm also concerned that it could take a lot longer than other services out there. But if you can make this service affordable, it could be every start-ups best friend.

Thank you HerbSCO for your comment. Both tester and developer rating systems will be integrated into the system.

It will up to the developer to determine if they choose to reward the tester with whatever compensation they choose. Obviously if giving a free copy away to a tester reduces their costs, they may choose to do so.

Hi noniesaft

Thank you for your comments and industry information.

"The first being acceptance of certification from the carriers."

First, Mob4hire uses the crowd to test mobile applications. Testing comes at many times during the process of development. To bring an application to a certifiying agent requires that the application is working on a specific device. If you do not know if your application is a "go" on a handset because you do not have the device, your costs will be very high because potentially your application will go back and forth between the agent and you to fix bugs.

"The second is the being able to ensure that all phones required are tested reliably."

You are not the first person to ask how do we ensure reliability. Quality will be key for the tester to continue testing if they wish. Developers will wish to have a very precise but complete test plan.

We will be certifying individual testers who have good rankings and have proven track records in mobile testing.

"But if you can make this service affordable, it could be every start-ups best friend."

noniesaft, when Mob4hire is up in September, we would like companies like the startup you are involved in to join Mob4Hire.

And noniesaft, you may be interested in joining the Mob4hire business to help advise us as we move ahead.

Thank you again!

I like the idea, you've obviously spent a lot of time on this, so great work! Having beta tested in the past, here is one thing that I know about beta test groups- you have the hardcore testers, worth great money, the average tester, valuable but not invaluable, and the slackers who thought it would be fun until it became boring, which of course, it does roughly 15-60 minutes after you start playing around and getting crashes which freeze your system! By what criteria are testers rated? How is their work progress tracked? Well done, fish!

hmm my phone can't run apps ahah.

But I like where you're going.

*correction: I like where you've GONE with this. ahaha. Little too late to be going to the tournement

Hey Bliss

"I like the idea, you've obviously spent a lot of time on this, so great work!"

I appreciate your comments and your support!

"Having beta tested in the past, here is one thing that I know about beta test groups- you have the hardcore testers, worth great money, the average tester, valuable but not invaluable, and the slackers who thought it would be fun until it became boring, which of course, it does roughly 15-60 minutes after you start playing around and getting crashes which freeze your system! By what criteria are testers rated? "

Testers will have a track record which the developers will be able to see. The developers will be able to see how many projects the tester has worked on and how they were ranked for each project. As well, a good tester will be transparent and will have an outline of their technical abilities. The testers will be ranked on a number of criteria including completed on time, and testing on all parts of the test plan(thoroughness).

"How is their work progress tracked?"

The tracking will depend on the test plan. You may find some developers that just want to know if the application loaded and fits in the screen. Other test plans may be very intricate (if they involve network data or LBS applications).
Some tests may be marketing based...ie surveys involving things like

Is the application intuitive?
Is the game fun?
Would you buy this application in this state.
How would you rank the animation graphics on a rank of 1-10?
Was the application slow?

I would imagine that some developers would like the tester to video tape their tests. (ie showing bugs as they occur)

"Well done, fish!"

Thank you Bliss!

Hey Patrick

Thanks for the comments....

My guess is your next handset will run apps...

Fish-
why not make Mob4hire a site that brokers crowdsourced beta testing in general with mobile apps being a specialized subset the apps? The mob4hire name itself is generic but you incorporated the phone in the logo... I agree with you that emulators can only get you so far and ultimately as an application developer you need real people using the app on real devices in real scenarios to flush out all the bugs like memory issues and performance under load, etc.

I have the same concerns as generic_idea_guy that you'd have to have a self-policing mechanism to review the reviewers to ensure the feedback you're producing is valid. The good news is if you abstract things well enough, I could see authors of web apps in general buying beta testers from your service to test their apps. You can solicit vaying levels of feedback and even have "bounties" for finding bugs and suggesting feature enhancements (ie. there's a difference in value between 'yes this app works' and valuable input on usability improvements etc). You might not even have to pay testers - it might be enough motivation for them to get a sneak peek at a yet-to-be-released app that some testers would test for free. good idea.

sean

Hey Sean (aka scrollinondubs)

Thank you very much for your comments.

"Why not make Mob4hire a site that brokers crowdsourced beta testing in general with mobile apps being a specialized subset the apps? The mob4hire name itself is generic but you incorporated the phone in the logo... I agree with you that emulators can only get you so far and ultimately as an application developer you need real people using the app on real devices in real scenarios to flush out all the bugs like memory issues and performance under load, etc."

Well, anything is possible ie make mob4hire bigger for other testing apps. However, I understand the difficulties in the mobile space and I believe the market is big enough for Mob4Hire at this time specific to mobile devices. I believe if the site was too general we would not get the response from the mobile developers. The forums will be geared to allow testing questions specific to the mobile space.

"I have the same concerns as generic_idea_guy that you'd have to have a self-policing mechanism to review the reviewers to ensure the feedback you're producing is valid.

Ranking the testers will help and will allow the developers to see their testing history.

"You can solicit vaying levels of feedback and even have "bounties" for finding bugs and suggesting feature enhancements (ie. there's a difference in value between 'yes this app works' and valuable input on usability improvements etc).

I love the bounty idea. For every bug you find we give the testers an incentive.

"You might not even have to pay testers - it might be enough motivation for them to get a sneak peek at a yet-to-be-released app that some testers would test for free."

It will be up to the testers to decide if they wish to test for free. With a bidding system, that would be possible. However there will be a minimum fee for the developers.

What about the existing websites & companies that all ready do this or offer this service?
GetJar.com < offers this service FREE to small developers! They have over 50,000 registered beta testers!
OnlineBeta.com
gamemobile.co.uk/beta-test-mobile-games.htm

Like I said before company's all ready do this and it is not that hard to get review models.

Thanks for the feedback Kevin_Cox

I am appreciative for the concern you have for Mob4Hire....

"What about the existing websites & companies that all ready do this or offer this service?
GetJar.com < offers this service FREE to small developers! They have over 50,000 registered beta testers!
OnlineBeta.com
gamemobile.co.uk/beta-test-mobile-games.htm"

I am familiar with both getjar.com and gamemobile.

Getjar is a good place to get some minor feedback re some standalone applications. The difficulty with their testing is it is entirely manufacturer/model based. For instance a Motorola V3i razor on Sprint is not the same as a Bell Mobility model nor is the same handset that is sold in Orange in Germany. They all have differences in software, memory and network connections. You may test standalone on one carrier model and it may work while on a different carrier model, it will not work at all.

"it is not that hard to get review models."

It is not difficult to get review models if you have the right connections at the manufacturers and the cash and connections in various countries to set up carrier accounts. This product will not be for every developer out there. If the developer has been around for while, they have had to spend tens of thousands of doallars to test their applications and they may continue to do so the same way. I think though they will looking closely to Mob4Hire in the future.

I wish you luck on your venture Kevin_Cox and I appreciate the competition.

Cheers Paul

"The difficulty with their testing is it is entirely manufacturer/model based."
Well, how is your service any different? Also, what stops them from just adding another field for carrier and putting you out of business? Because they all ready have a user base of every make and model for phones and headsets.

Hey Kevin_Cox

Thanks again for your constructive comments...:)

"The difficulty with their testing is it is entirely manufacturer/model based."
Well, how is your service any different? Also, what stops them from just adding another field for carrier and putting you out of business? Because they all ready have a user base of every make and model for phones and headsets.

Any changes to potential competition are out of my hands. (as is any website or application with a long leadtime to get to market)

What Mob4Hire is doing to be different is to allow developers to choose who their testers will be rather than a random choice of whoever wants a free game or application on their handset. The testers will also pick which applications they want to test on. The testers will get paid by the developers and both will will benefit from the testing.

Again I wish you luck Kevin_Cox.

Paul

I think this idea is brilliant, especially if you expand it to any type of software testing. Every new developer needs a quick legion of followers who will rate the product and spead the news. Same concept as the book, "The Tipping Point". I think Mob4Hire could quickly get a product tested and into the marketplace to generate revenue.

Hi Fish99/ I realize this is late, but I'm going to post them anyway and see who responds!

For the TOC Q&A:
Q2: Just wondering how you intend to handle intellectual property? Most companies view IP as super important, and many developers may want users to sign NDA's before testing software.

That and they all ready can select there own beta testers and usually do all ready through there own applications. "NDA's" I don't think would be that big of a deal they are very common in the testing industry.

Sorry to answer you question I should have let fish99 add his two cents first.

"I think this idea is brilliant" Thank you Collabori!

"Hi Fish99/ I realize this is late, but I'm going to post them anyway and see who responds!

For the TOC Q&A:
Q2: Just wondering how you intend to handle intellectual property? Most companies view IP as super important, and many developers may want users to sign NDA's before testing software."

Thanks TravellingGuy for the question.

I will answer in text here but will make a video as well.

Q2: Answer

The arrangement between the developers and the testers will be up to the developers. If the developer requests an NDA, it will be up to the tester to decide if he or she wants to sign the agreement. If the tester says no, the tester will not likely get the testing work. It is quite simple.

Not all developers will require NDA's but some will likely wish to use them.

For those last minute voters please check out the commercial video at the top of the page. It will give a good idea of the scope of the market and a big problem with the market....as well as a solution!

or go to http://www.youtube.c.../watch?v=odV9A_H4uRI

Hey Fish99.

Just checked out your commercial video. Good work! I thought it was simple, yet very effective, and I loved the choice of music.

ScrollingOnDubs and Fish99 were having some discussion further up (Sept 2nd) in this thread about "self-policing mechanism to review the reviewers to ensure the feedback you're producing is valid".

Seti @ Home does not consider a workload complete until another random user has looked at the data and come up with identical results. The 2 users are not going to know who each other are, so its impossible to collude.

Maybe any bug detected is fired off to 1 (or 2) other random testers for them to verify. Or maybe they don't know what they're looking for. Just feed them in the original testers scenario and see if they report back anything unusual.

Just a thought on that.

Great Video

congrats Mob4Hire team :)

"For the TOC Q&A:
Q2: Just wondering how you intend to handle intellectual property? Most companies view IP as super important, and many developers may want users to sign NDA's before testing software."

Video response. http://www.youtube.c.../watch?v=mTypDgPnrd0

Hey Gord

"Maybe any bug detected is fired off to 1 (or 2) other random testers for them to verify. Or maybe they don't know what they're looking for. Just feed them in the original testers scenario and see if they report back anything unusual."

Thanks for the feedback. That may a good idea down the line when there are strange testing situations where the results are intermittent.

To TravellingGuy, Gods_Light and Fossiloflife

Thanks for the feedback!

Paul,

"That may a good idea down the line when there are strange testing situations where the results are intermittent."

For Seti, its useful flagging cheaters. If you fabricate your results, your results won't match the other random tester's results. Of course what is being returned in Seti's case is a chunk of mathematically derived data, not a simple boolean yesError/noError (or messy text based) response. So its suits Seti better, but maybe there's some way of using the approach to detect lazy participants who aren't really searching for bugs, and just respond that everything is ok. By forcing >1 participants to check same process (on say same platform), you'll know stuff isn't getting missed 'cause of a slacker.

Hey Gord

It will really be up to the developers and their test plans. Some developers will want a random run though the application. Some will want step by step instructions and deliverables based on each step.

I would like to thank all the members who have voted for Mob4Hire so far!

I wish Game all the best in his venture and I look forward to the competition.

As an update, I would like to mention that Mob4Hire is using only Cambrian House members to complete this project.

We should be up in alpha next week. We will be launching a prize for Mob4hire signup. An iPhone or $500 will be given away randomly to those people who sign up as testers. Developers will be given a $200 credit towards their first testing project.

Out goal will be to help developers in not only technical aspects of testing but also marketing aspects. For instance, is this game sticky? and does this application have a good graphical user interface (GUI)? Outside testers can tell them.

The key here is it will be a win-win for all parties involved.

The developers will win because they will get testing done on the handsets they require.

The testers will win by being a part of the process and will get paid for their efforts.

Mob4Hire will win from getting the right people together for this task and will be paid for doing so.

This is a "pain -killer" application. There is pain on the developer side...

Mob4Hire can solve their pain.

again a hearty congrats to the Mob4Hire team!

CH staff ask questions!
http://youtube.com/watch?v=RwSNNgYs_qg

Hey I'll post these in text form shortly, but no need to wait on that if you want to respond now. Yes, I'm posting an identical YouTube link under both ideas. One video covers each matchup!

fish99, here's a text copy of our video questions...

Q - It may be easy to catch critical errors which result in crashes or error message to pop up. But how can you communicate expectations that might not be met due to subtle differences between different phones. For example, if your writing an application called "Lego Junkie", where lego blocks are shown animated, as they fit together to build say, a replica of SCO CEO Darl McBride. Now the button which starts this animation is the "Assemble Model" button. But on the cellphone I'm testing on, I've got a smaller display. And instead of seeing a button called "Assemble Model", I just see a button called "Assemble". Or on an even smaller phone, like the one in Zoolander, all that's shown on the button is "Ass". All 3 buttons make sense, but unless the tester has multiple testing platforms to try the application on, there's no indication that anything is wrong. For a tester who is just using their one and only personal phone, how can such subtle errors be caught?

Q - The mobile phone industry is continually moving towards write-once-run-anywhere platforms, nGage being one such step forward in abstraction. Aren't you solving a disappearing problem? Surely the phones of tomorrow won't require a different code base for each handset, and testing of each and every phone configuration?

Q - Won't internal QA be tough to compete against? For a small development team on a tight budget, wouldn't it be more cost effective and offer tighter control to simply test internally? And the bigger software firms specializing in cellphone applications already have testing solutions, and their own economies of scale.

Q - I can't get data to work on a Telus Blackberry someone gave me without signing a contract with Telus. I just want to pay as I go, but I want to check email too! No, in fact don't even talk about it. Makes me angry just thinking about it.

Q - It may be easy to catch critical errors which result in crashes or error message to pop up. But how can you communicate expectations that might not be met due to subtle differences between different phones.

For example, if your writing an application called "Lego Junkie", where lego blocks are shown animated, as they fit together to build say, a replica of SCO CEO Darl McBride. Now the button which starts this animation is the "Assemble Model" button. But on the cellphone I'm testing on, I've got a smaller display. And instead of seeing a button called "Assemble Model", I just see a button called "Assemble". Or on an even smaller phone, like the one in Zoolander, all that's shown on the button is "Ass". All 3 buttons make sense, but unless the tester has multiple testing platforms to try the application on, there's no indication that anything is wrong.

For a tester who is just using their one and only personal phone, how can such subtle errors be caught?

Transcript of answer on video

Hey Gord

Thanks for the question.

What you are describing effectively is GUI testing..

For example on smaller handsets does the graphical interface make sense? Do you have to adjust the words used or icons to make the application work on a smaller handset screen? Is the resolution of the screen good enough to understand what you are supposed to do next?

The application developer has to wear a lot of hats. They have to make sure the code is ready for a specific handset. They have to make sure the application runs the way it is supposed to. They have to make sure the application makes sense to the outside user. They may be cranking out 30 versions of the code for a carrier.

Mob4Hire will allow the developer to ask what ever questions they wish. The developer may decide to test on whatever aspect of the test spectrum they wish. One of the questions the developer may ask on completion of the test is was the application intuitive and ask the tester to evaluate it on a basis of 1 to 10. Not many people look at the help screens on the mobile phone based on the game or application they are using.

The use of an outside tester will give a quick indication if the application is intuitive. The use of inside testers who already know what the application should do can be quite biased.

Video reply....http://www.youtube.c.../watch?v=fUBLREbBOrU apologize about the slight lag in voice.

Video question #2 - Won't internal QA be tough to compete against? For a small development team on a tight budget, wouldn't it be more cost effective and offer tighter control to simply test internally? And the bigger software firms specializing in cellphone applications already have testing solutions, and their own economies of scale.

Hey Greg

Thanks for the good question.

You know the reason Mob4Hire was started was for the small development team on a tight budget.

Imagine you are a small developer in France. (imagine that Mob4hire is not around) You decide to build a network application..lets say a online contact manager. A large US carrier shows some interest but wants to see the application on their phones.

You have to crank out code for the project. You have to make sure 30 different handsets with different screens sizes, different cpu’s, different screen resolutions, and different buttons on the handsets will run the application.

First because you are a small application developer, you do not have an office in the USA. So somehow you need to get the handsets to test on and you do not have them.

You spend weeks trying to get free phones from the carriers but they are too busy to answer your calls or they have run out of test phones. You talk to the manufacturers to get the handsets. Some of them want you to pay to join their development website.

Somehow you manage to get all the handsets you require. You load the software but because you are in France, you cannot test the application because you are not running on the carrier network. Not all carrier networks are alike.

So with the 30 handsets you send an internal team of 3 people from France to the USA to test the applications. Cost 2000 per person travel and other costs. $6000 in total.

Now to get on the network you want to test on, the testers go into a carrier retail store to get their phones on the network. The carrier has interest in the project but does not want to pay for the monthly fees to allow you test the application. So you get 30 one year subscriptions for the phones. Cost is around $800 a phone. $24000 dollars in total.

Great now after 2 months you have your working phones, and you have a team of your coders testing the applications in the USA.

The testers are the internal coders that wrote the application and the translation from French to English was not very good but the coders really do not know this. The translation of the help screens does not make sense in English. The coders also know the application inside and out and do not find any bugs because they have been following the same test pattern by rote.

This is not a cheap or good solution for the developer. Cost not including labour is in around $30,000. The test was biased and the help screens are still poor. It took 2 months to test from the start.

There are other testing solutions that may be moderately cheaper but still in the same ballpark.

Now let’s look at the Mob4Hire solution.

The developer puts out a call for testers on the specific network on Mob4Hire. They require the test plans done within a week. They find the testers they need. The testers are not biased and will tell the developer that the help screen does not make sense, and they find a number of bugs, the inside testers would not have found. The testers on average charge $50 each for their work.

After it is done, they have tester bug information, marketing feedback and they are completed in a week. Total cost $1500.

Large companies as well, will get good marketing feedback on their product through the use of marketing testers. Ie Would you buy this product?, Was it fun? Was it intuitive? This will still be cheaper than going through focus groups.

video answer to come.....

Q3 - The mobile phone industry is continually moving towards write-once-run-anywhere platforms, nGage being one such step forward in abstraction. Aren't you solving a disappearing problem? Surely the phones of tomorrow won't require a different code base for each handset, and testing of each and every phone configuration?

If you are in the mobile development field, you will quickly find out that the write-once-run anywhere theory is not true at this point.

In fact it has got worse in the last few years ….

The problem lies in cost. Each carrier gets smoozed by different manufacturers of handsets. The carrier goes with what they can sell this quarter. The CEO’s of the carriers are driven by quarterly results.

The handset operating system makers also are driven by the same model…
Therefore, for example, Nokia drives Symbian sales and Microsoft is trying to sell their Windows mobile and Sun with J2ME and Qualcom with Brew.

You may get some better results under one platform. (ie Nokia) but there are few carriers who are going with one handset maker only. They have to manage their risk.

Do a google search for “write once run anywhere mobile” There is no consensus.

I do realize that somewhere down the road, there may be movement to the write once run anywhere model, but right now, the application developers are hurting.

some comments for the website:

http://www.mob4hire.com

-- put in some images that explains how the process works
example:
the "how it works" images on http://www.cambrianhouse.com

-- And/or put scenaiors how the process works
example:
http://www.jayparkinsonmd.com/Wrist.html

-- remove forums, it makes the site looks cheap. Specially a generic forum format. It makes the project looks like you cannot host your own content/code...I dont know maybe just my personal opinion.

-- Add a Blog. All successful businesses do this effectively (from Google to successful MD's in NY ---> refer to example above :)

-- you need a "drop down" section for different brands/phones. However this can be quite a task. However if you implement it in a nice way, then you are going to attract business

-- But I personally think the focus should be on "selling the idea" through your site. And to be able to capture the imagination/attention in a realtively short time. I think two of the ideas that I mentioned in the beginning are going to boost the popularity/selling-the-businee-model for your site immensely. You do not need fancy/shmancy flash content to be doing this. Just the right business model/an attractive mechanism to deliver it through your site/**most importantly you need tools on your site that enable customers to do what you want them to do/ and then you have to have a benefits/royalty section.
Drop me a line sometime of your proposed business model, if you need more feedback. I must apologize I havent had the change to go through all the comments.

Paul,

Think *big*

Think :
- Alliances with cellphone manufacturer's and carriers
- Think virtual devices/phones and effective mechanisms to gauge and assess the performance[a]

I am thinking something along the lines of:

-- you sign an NDA with the different cell-phone manufacturers
-- they give you the code for the OS that is supposed to run on their phones
-- you take that code, implement it on your site with virtual interfaces that simulate real-life cell-phone usage
[a]:
-- then you implement tools like the "rational robot" (automated QA). Hence you can have a semi-automated engine. Where warm-blooded bodies drive the front end of the testing.
- 1) http://www-306.ibm.com/software/awdtools/tester/robot/

Got to brain storm this idea a bit more. Primarily how "true user" experience is going to be implemented when it comes to the "voice" aspect of the cell-phone usage.

Things that work for this idea:

- Cell phone manufacturers spend a lot of money testing their own product. If a bug is found for any given reason, which cannot necessarily be fixed with a software update..then the manufacturer ends up dishing out millions of dollars in damage fee to consumers/carriers they sold the device to
--> if a product such as mob4hire, can take the manufacturers worries away, then they are going to come flocking to you for having their product tested out

Things that do not work for this idea:

- Manufacturers testing their own product. Usually cell phone manufacturers have a lil army unit within their company for this specific purpose. (unless it is automated...which I am hoping is usually the case)
- Confidentiality: carriers certainly do not want the information about a new launch/new device leaked prior to the actual launch
- in that regard (confidentiality) , testing for a product itself would conflict with the confidentiality terms and conditions (unless you have a solid alliance with the manufacturers)

Now reading all my comments, you must be thinking that i got the idea all wrong ...that this is supposed to be about "testing applications on different mobile handsets". But I guess we got to take a step back and ask ourselves? Is that really where the money is?

If your answer is Yes, then would it be prudent to do some market research in that regard. (I can elaborte a bit more on this if you want)

On the flip side, is there more money to be made for the different points I mentioned above. If that is the case, then you should make this project really BIG

nice logo/ads.....whats the latest on your team?

Hey Gen

Thanks for the comments!

I will contact you.

Hey Mi_amore

we are in alpha (registration only) at http://www.mob4hire.com.

and thanks for the compliments.

Paul

http://techvibesblog...x-january-2008-beta/

Mob4hire makes top 10 list for Alberta startups from techvibes...:)