I am happy to say that it works as good as I wished it to – though there are always more features I can add.
My app pulls information from news.ycombinator.com (hacker news) and displays it succinctly. It doesn’t seem too hard but I didn’t want HTML scraping and parsing and storing on the phone, and I wanted a LOT of data. My original plan was to scrape 3,000 links from that website every 10 minutes and store it in a database somewhere (??). I had the idea of using the CS345TEAM5 database from my database class, and I got Professor Morrison’s approval on that… the only bad thing is my data will be wiped at the end of the semester.
So, I learned PhP a bit for this project (had a little bit for a DB assignment, now a little bit here). When testing my big scraping file, I was blocked from the website. I realized that the website blocks any ip that pings them too much… maybe they thoughtI was DoSing them, so I changed my plan to scrape 300 links every hour… lighten up the pinging. Also, when scraping, I would scrape one page a minute for 10 minutes.
The website still blocked me. I thought I would have to abandon all hope and choose a different website, but, when scraping 5 pages from their site every hour they never block me. My data isn’t as ambitious as I originally hoped but it’s probably better to not transfer as much data to a dinky phone, so hey, it works out.
So my app hooks up to an on campus website that I had access to yet from the DB homework, pings a script I wrote that pings the database to return data in JSON format. Awesome!
Then I noticed that the database was storing some crazy values – Euro signs where they shouldn’t be (i.e. at all) and other weird characters. Apparently, hacker news uses UTF-16 encoding, whereas the php command that scrapes a website translates everything to UTF-8, which broke some characters. I dropped the database a few times when it had little data and changed my script to parse and replace the misc. characters before insertion. As I noticed more later on in my app I would just search the DB and replace with a select statement.
…and lastly I noticed that when the php script uses json_encode, it also messes up some characters. I don’t know if, if I didn’t change the database, it would take the cryptic letters and put them back to normal, but I think that may be the case and I originally wasted my time ‘fixing’ everything.
I started the app with the intent to get a graph in there somehow and I found some android graphing API’s online. I decided on using afreechart.jar (http://code.google.com/p/afreechart/) because that package has the best looking graphs for free.
There is little to no documentation on it, and figuring out how to even get a graph working was ridiculous. I eventually found some code that I broke down and down until it had just a few lines to put 3 points into a graph… and I learned. Awesome, again!
Back to my app; now I have an activity that has a bunch of links on it, and an activity based off of that that displays a lot of information and a graph about the ebb and flow of the link on hacker news. This is most of what I wanted.
There are a few features that I didn’t implement that I don’t know if I really want to anymore (because it would be distracting): tracking which links are new, tracking what links changed the most in points since last checked, and having more graph features.
The graph features are worth looking into, the other two, not so much.
The last thing I implemented into the project because I thought it would be cool is the ability to search for old links. This, again, pings a php script I wrote.
I decided to write this in both Android 4.0, which was the only android sdk platform I had when I started, and in Android 2.3.3, so that Chris could test the code on an actual device, because I love this app and it’s freaking cool. Android 4.0.0 looks amazingly better, though.
My app requires the internet so I check for it in every activity besides the home one – the user isn’t going to get internet data without an internet. I also handle orientation changes: in most activities I allow rotation and keep the data, however, in my activity that holds the graph I do not allow rotation (graph wouldn’t look good).
Also, I designed the icon for this, in low medium and high resolution, and it looks really cool.
Everybody seems to be posting code but I didn’t really find anything on the Java side the most complicated… so here’s some of the php scrape script that gave me troubles for 2 weeks:
while($i < 5 && $i > -1) {
$startParse = 0;
$endParse = 0;
$ups;
$nextStartParse = -1;
echo "scraping...";
$source = file_get_contents($baseurl . $link);
echo " parsing...\n";
while($nextStartParse !== false) {
$startParse = strPos($source, $searchForURL, $startParse) + strlen($searchForURL);
$endParse = strPos($source, '"', $startParse);
$parseLength = $endParse - $startParse;
$link = substr($source, $startParse, $parseLength);
$nextStartParse = strPos($source, $searchForURL, $startParse + 1);
if ($nextStartParse === false) {
if ($link{0} !== '/' && substr($link, 0, 4) !== 'http') {
$link = '/' . $link;
} else if (substr($link, 0, 4) === 'http'){
$i = -100;
}
echo $link;
break;
}
$k++;
echo " link number " . $k . " ";
if (substr($link, 0, 4) === 'item') {
$link = $baseurl . '/' . $link;
}
$startParse = strPos($source, '>', $endParse + 1) + 1;
$endParse = strPos($source, '<', $startParse);
$parseLength = $endParse - $startParse;
$title = substr($source, $startParse, $parseLength);
$title = str_replace('“','"', $title);
$title = str_replace('’','\'', $title);
$title = str_replace('â€','"', $title);
$title = str_replace('\'¦', '...', $title);
$title = str_replace('"¦', '...', $title);
$title = str_replace('"“', '-', $title);
$title = str_replace('', '', $title);
$title = str_replace('"˜', '\'', $title);
$title = str_replace('"”', '-', $title);
$origStartParse = $startParse;
$startParse = strPos($source, $searchForComhead, $startParse) + strlen($searchForComhead);
$endParse = strPos($source, ')', $startParse);
$parseLength = $endParse - $startParse;
$abvUrl = substr($source, $startParse, $parseLength);
if ($startParse === strlen($searchForComhead) || $startParse === false || $startParse > $nextStartParse) {
$abvUrl = null;
}
$startParse = $origStartParse;
$endParse = $origStartParse + 1;
$startParse = strPos($source, 'id=score_', $endParse + 1);
if ($startParse !== false) {
$startParse = strPos($source, '>', $startParse + 1) + 1;
$endParse = $startParse + 1;
while(is_numeric($source{$endParse})) {
$endParse++;
}
} else {
$endParse = false;
}
$parseLength = $endParse - $startParse;
$hopefulUps = substr($source, $startParse, $parseLength); //may be empty.
$ups = null; //even though this is converted to a string, I want to make sure it doesn't containt characters yet
I had so much trouble getting that working completely, but it’s awesome now. I’ve had it set up hourly on a scheduler on my computer running nonstop, which means I can’t turn off my computer.
One glaring thing I could clean up is fix how many times the user needs to ping my database. I have the person pinging it every time they search for links even though I only update my links every hour… this should be fixed. Otherwise, I tried to code this as securely as I could.
And finally, one cool ‘trick’ I did is that, when designing my Search activity, I have it spawn my normal link-displaying Activity and just reuse all the code in it. It saved me from having to specifically write an entire new complicated ListActivity to do roughly the same exact thing. Less of a trick, though, and more of foresight.
Was fun, yo.
]]>
My last app scraped the HTML of news.ycombinator.com and saved the results into the phone. Every time the app was reloaded, the app would show the biggest changers (links that got the most points).
I’d like to pull this database like feature off into an online database that would automatically scrape the html every 10 minutes and do all the biggest changer/data storing over there. Then, on the phone, I’d like to have multiple screens: a list of all the current, all the new, all the biggest changers. Each of these would be on a separate page. The tough part about this is going to be getting a scrollview on the same screen as a gesture listener. I tried doing this in HW3 and it proved difficult so I left it for a future experiment.
I’d also like a little button on the bottom right that can be clicked to bring up a search box to look through links. This is mainly a UI feature and shouldn’t be hard to implement in the database.
I’d also like to have more information per link and a prettier interface. For more information, maybe click on the adapterview item and then it’ll take you to a detailed link page with the first few comments (or a link to the comments) and maybe a graph showing how fast the link is gaining points (per every 10 minutes).
The last thing I’d like to implement is a pretty UI. The backend is going to be difficult and I’d like to focus on that first, but a nice UI is always… nice.
The biggest problem I have is using an online database… I don’t know where I’d get this. I was hoping to use some school servers but who knows. Maybe I’ll have to come up with a different idea.
But if I do get the online database I think this could be a complicated yet cool project that could be extended to other sites that just list news articles.
]]>My app connects to news.ycombinator.com (hacker news) and pulls the first 5 pages of top stories.
I thought that this would be very hard and the first thing I did was pull all of the wevents code and try to modify it to what I need. This didn’t work and, after 6 hours I started over completely. This project taught me yet again that building from the top down is not a good approach. It’s one of my bad habits, but everything always looks so simple from the outside.
One of my first main problems was getting the service to be called. Apparently I put my service declaration outside my application declaration in the manifest. That took me a while to see.
Then I had to realize that I need to set the receiver in my activity.
Then I had to figure out how I would parse the data from hacker news — what I ended up doing was looking for id’s and class declarations in the text and using a big stringbuilder to get multiple substrings. I pass around a big string that has all the links concatenated with ~`, which I break apart using the split method.
I didn’t want to use a database because my original goal was to have the upvotes from one time compare to upvotes from a previous time and only display the biggest changers. So, I stored my huge link string in a file, which I thought would be fast, and I build an array adapter using the split method (again). Clicking on items in the adapter (which have the number of up votes and the article name) opens a browser to the article. This was an easy part to implement.
Apparently using files is very very slow in the emulator and after making a file testing became very hard. My file is only about 20000 bytes big, which should not take long at all to load, but this takes about 30 seconds to a minute to load or more.
An interesting part about my project is that I scan the source code for a link to more news stories and I issue another request to open that page too. I all, I swipe the source from 5 hacker news pages.
I would have attached a picture to this but in the last two hours leading up to and a little bit after midnight my emulator and eclipse began acting very buggy and the emulator wouldn’t load and wouldn’t shut down and eclipse wouldn’t really update the code. A computer restart and everything didn’t clear this up so, thankfully my code was pretty well tested but there might be some bugs that the final hours made impossible to fix. I couldn’t test offline mode because, even when offline my emulator would load data from the internet and, when I shut my internet off the emulator wouldn’t load. Also I didn’t get to testing configuration changes and there are a few other areas that I couldn’t test. For some reason my emulator hasn’t been working right at all. I used Android 4.0 on this project and ever since I upgraded everything to the current releases (Tools 15 and such), my emulator and other testing has been a little off. So who knows, I think the concept of the hacker news articles is cool and I think what it does currently is a really cool base but, more can be done and I wish I could have done it.
]]>Originally I had grand plans of querying the web for values and allowing the user to search for a location or click and get the location name. This would have been done using something called Geocoder but apparently Geocoder doesn’t work on an emulator. I tried the workarounds online but, as I wasn’t working on my own computer, I didn’t have privileges to edit a file that needed edited (dang!) so, I gave up with the Geocoder after many hours of work.
One thing that’s lacking in my app are any directions so, they definitely need to be explained here. I have an on long touch listener on the list activity that’ll take the user to an edit screen where the user can edit/see the name and description of a place. Simply clicking on a place will take the user to google maps with centered at the location with a triangle pointer (that I designed!) above it.
If the user straight up clicks the “Go to maps” list item then it’ll take the user to the last looked at spot but, zoomed out more.
On the maps screen the user can hold down for a second to add a new location. The user can also touch an existing location to get the name of it.
I had some difficulties getting the adapter to refresh after deleting an item in a separate activity, so I eventually decided to loop through the entire adapter and see if the returned ID matched an adapter ID and, if so, remove the item (check the code that you don’t have, that sounds confusing).
I’m a lot more pleased with this app than the last one just because the user interface is so much smoother… the last one was just atrocious. The only *glaring* bug I can see is the one where a user adds a new place. It automatically triggers if the user holds down on the screen for 1 second; I was trying to get it to go based off of how far the user moved (event start vs event end) but there is no way of tracking start and end events. I tried creating an inner class that would track the start and then sleep for one second but stuff just wasn’t working so I left this detail to be fixed at a later time.
I used the Rattler application as a base for all the database stuff but I had to branch out and figure out problems specific to my part.
Was a good challenge; I give this two thumbs up.
]]>It’s basically just like everything else that’s been posted… at least 5 questions and such.
For my app, if a user gets a question wrong, the user can retry, try again later, or skip the question entirely. When the user is done with the game he or she can click to be redirected back to the main activity.
A user can also add questions with the Add button on the first page. The are validation things where if the user tries to input a question that has been put in before or questions with blank text then it’ll stop the user update the hint in the input box (informing of the error).
The main problem I had was keeping track of the Database I used. I was simply too busy to look up how to implement an actual database and persist data, which is the method I was really holding out for, so my database tracking involves ArrayLists of Strings. It really wasn’t fun. Towards the end (if you look at my post time you can see what I mean) I was just trying to get the database of strings solidified with multiple activities that I began to write spaghetti code. It’s really not pretty. Definitely could do a better job in a future revision implementing SQLite whatever.
Also, making nice looking layouts is tricky. Definitely something I want to look into more in the future.
Regardless, it was a great exercise and forced me to learn the hard way the issues of passing around data without any permanent files.
]]>
It’s got the basic functionality of a simple calculator but not much more. Learning to use the calculator is basically intuitive. To get to a menu with a caret for power for exclamation for factorial, one simply presses menu. The keyboard doesn’t work the best, though, as the % button is assigned randomly (as well as to the % over 5; if the user presses R, the percent sign is shown).
Basic calculator functions are supported, but only the very basic. Imaginary numbers aren’t supported. The screen allows a user to input as many characters as he or she would want, which is somewhat dumb, because a person using a simple calculator isn’t going to be doing billion number calculations and isn’t going to need all the space. Also, addition is all done on one line instead of hitting (+) and then having a new number pop up when one types.
Some functionality doesn’t really make sense. After doing a calculation, there is a way to get back to the previous entered input but it’s not very intuitive and I couldn’t find a concrete way of doing this in my few minutes of playing around.
The app only exploits the basics of a mobile platform, i.e. the touchscreen. There isn’t much more one could do with this, however, shaking for erasing would be a nice feature.
I imagine there are other more advanced calculator apps out there but, seeing as I don’t have a smartphone, (or any android device), I can’t really test these.
There really was no stress testing for this app. Large calculations would lead to E notation and then the infinity sign being displayed (as seen in the first picture) .
Because this is a stock app, money doesn’t really play a role. However, I imagine that a more functional app from the store would cost money. There is no way to make a profit in the calculator unless you showed an ad on load or on close, so charging for an amazing calculator would be a good idea.
Really, I wouldn’t only use a calculator like this for finding out store prices or tips or anything. I would never use a calculator like this for serious math work, and I imagine that this app isn’t designed for serious math, anyways.
-Travis Bischel
]]>