4 years of development - Fourth generation

Fourth generation - Server no more

With each generation, the release was delayed by new problems on the server side and difficulties to keep the server running reliably. I realized that it was not feasible to maintain both the app and the server in the long run. I decided to stop using the server and started working on making the app work without it. By the time I shut down the server, it had 268 193 unique tiles (PNGs) for different zoom levels, over 3 gigabytes of raw data and over 1 million unique Wi-Fi networks. It took a lot of time to process each upload on a single core virtual machine (DigitalOcean). Just as a note, I don’t regret making the server in the first place, it was a great learning experience. To summarize the many changes that were needed to make most or all of the server features work on the app itself:

• goodbye server side:
  • authentication,
  • feedback,
  • upload,
  • shared maps,
  • shared statistics.
• replace JSON with a database (SQLite),
• implement challenges,
• rewrite statistics,
• create a local map tile generator.

These few points mostly cover what had to be redone, which if we add it up, is almost the whole app. Not only that, but this update also brought many improvements, like more detailed statistics, heatmaps, import/export, better style updating, new options, refactoring in many places, tracker overhaul, activities, modularization and more. There were probably not many parts of the app that were left untouched.

From server to phone

One of the good things about the migration was that many algorithms could be reused for the client side logic. Kotlin and C# are quite similar for algorithms, so code can often be just converted to different syntax. However, not everything could be ported. For example, map visualization had to be completely rewritten.

Map visualisation

The original map visual was designed for server-side processing and fast incremental updates. It used a square grid and a caching mechanism to display the data. However, this design was not suitable for mobile devices. It required a lot of CPU and memory resources to compute everything in the background. It also wasted disk space, as each tile occupied at least 4KB, regardless of its actual size. Moreover, it drained the battery power faster than necessary.

Therefore, I decided to change the map visual to run on demand. I used heatmaps instead of square grids. Heatmaps are more efficient and accurate for local processing.

Heatmap generation

Heatmaps are a great way to visualize data, but they can be expensive to compute. I searched for the best solution and found very efficient algorithm written in C, which I then ported to Kotlin. The method I used to create heatmaps was based on a stamp¹. A stamp is a small image (the default size was 9x9 pixels) that shows how the surrounding area of a point will look like.

The algorithm then iterates over all the points which belong to the area and stamps them on the tile. This results in fairly good heatmap generation. It looks pretty well, but it still has a number of issues.

Time awarness

One problem with using stamps is that they do not account for time. If time is ignored, staying at one spot and recording one location every second can make it the hottest point on the map. And visiting the same spot five times a day can make it seem very cold compared to that. I solved this by adding information about the last update to each pixel². This might seem like a perfect solution, but it is not because it causes a problem with blending (you can see an example below). This problem happens when a stamp can only be partially applied because some of it is too old. Throwing away the whole stamp is not a good solution either, because you lose information. I tried to fix this by using a blending function³, but the problem still exists. I experimented with different solutions, but most of them made things worse. The only proper solution I could think of was a better blending function that would blend the data better, while still considering time.

Tile handling

Another problem is how to handle tiles. What you see on the map is actually made of multiple images that blend together. My solution to this problem was to draw each tile slightly bigger than it should be⁴, so that the edges are drawn correctly even if some points are not visible.

Transparency and color

When blending two colors, transparency and color need to be handled differently. This is due to how humans perceive colors in general. If you use the same algorithm for both, the colors might not blend nicely and create some unexpected colors. I decided to solve this by using different blending functions for transparency and color⁵.

Achievements and challenges

Achievements were a nice feature of the server, but they were too hard to implement from scratch. They also required a lot of computation, as every achievement had to be checked against every relevant data point. For instance, Wi-Fi achievements might have to scan and compare 200 Wi-Fi networks every few seconds in some locations. This would drain the battery faster. Even with some optimization, it would still be computationally intensive. A better alternative was the Challenges feature. Challenges were similar to achievements, but they had a time limit and only a few were active at any given time. This reduced the computational cost and did not depend on the number of challenges available. They can also take some time to compute, but this is done at the end of each session which reduces the battery drain, by doing all the intensive computations at once.

Statistics

The application uses statistics to provide a summary of your session. The original concept was not well-designed, so I decided to improve it in the 4th generation. I used a system that I had created before for tracking - consumers, producers and raw producers. Raw producers are responsible for loading the raw data from the database. Producers transform the raw data into more processed data, such as reducing the number of locations. Consumers take the producer data and convert it into the final format that can be shown to the user. This system has several benefits. It is expandable, maintainable and cacheable. It is expandable because adding new statistics is easy, you just need to create new consumers and producers that follow the proper interfaces. It is maintainable because the system is not very complex and does not get messy with new additions. However, it might be a bit harder to understand than the previous chaos, because there is some logic behind it, while the chaos had none and just did everything in a “single function”.

Cacheability is also an interesting feature, but it is not clear if it is really useful. Caching inherently adds some overhead to the initial processing of the document. This could be improved by caching in the background. But how often do you revisit your sessions and how long do you have to wait? In a commercial project, I might have skipped this feature, because it does not add much value. But from an experimental point of view, it is a nice feature to have and that’s why I implemented it. In its current form, the overhead is not too bad and it makes subsequent openings faster. It is also important to be selective about what to cache, because sometimes it is faster to just recompute everything than to store it in the database.

Conclusion

The Android app has gone through a lot of development and still remains functional even without the support of its server. It has some impressive features that make it stand out. The app itself has about 26 500 lines of code, plus a few thousand more in its own dependencies that were built for it. What does the future hold? Maybe a new operating system, because Android is hopeless under Google’s control. Thank you for reading a story of Advention. There will be one additional part to this series that will talk about interesting features of the application.

Takeaways

I compiled a shorter summary for developers who would like to take away something from this.

• Google Play Services can cause instability and compatibility issues, so use them with caution. (They may break your app with unexpected updates)
• Google Play is not reliable for crash reporting, so use a different service like Crashlytics or similar.
• You need to keep your app updated and maintained, because new Android versions may introduce new problems. This is especially true if you use features that consume a lot of battery (Google seems to be cracking down on those lately).
• Android does not support customizing styles very well, especially if you want to let users choose RGB colors.
• Interfaces are very helpful when you need to change something, because they can isolate the changes and prevent you from having to modify a lot of code.
• Modules in Android are not very mature yet (at least at the time), and they have many limitations and drawbacks. (But they are necessary for larger projects) Avoid using on-demand dynamic modules for now.
• Allowing users to choose any background image is a nice feature, but it has some trade-offs and challenges. Most apps should stick to light and dark themes only.
• Android has many flaws and bugs, and Google does not seem to care much about fixing them or improving the platform.
• Allowing users to pick their own language is very frustrating, because there are many things that do not work as expected. (This should improve with Android 13)