We have brand new infrastructure built out and new servers. Our old provider was failing at keeping up with traffic and balancing the load, which is not ideal for our users. We were encountering outages randomly and having DNS issues with servers getting deleted. Our team was loyal and that was to a fault. We gave our old provider months to try to figure out this issue, and it came down to them not being able to figure anything out.
So we looked for a new provider to host our servers and apps. We looked at a bunch of them and found one that had great documentation and could handle our growth with zero downtime. At least that's what the documentation said.
We did setup a meeting with them and they answered our questions about the different services they offered and what the benefits of going with them over other providers were. They answered all our questions, and we finished moving over everything about 3 or 4 days ago. We rebuilt everything from the ground up and moved all the data, including this very blog. Our primary servers were moved with zero down time. This was a big task that had to be done to give our users a better experience in the end.
This server infrastructure will auto scale up and down as traffic comes in and should not need much human interaction to keep running. This will help us with getting more features out for the apps and improving our code that powers everything.
We have also added more outage detection and latency probes to our monitoring infrastructure to know if anything is wrong or not working. We will be alerted from multiple sources if there is any kind of problem or outage.
Almost all the BigBearApps now monitor connections to the BigBearNetwork and will say up top if they can not connect to the BigBearNetwork. Just click on the banner up top of the app to go to our status page that is updated about every minute or so.
If your phone disconnects or does not have internet, it will display a banner stating that it is unable to connect to the internet.
To provide our users with the greatest experience possible, we are continually working to improve security and make our apps and backend as fast as possible. We increased server speed and made various security upgrades with the introduction of BigBearOS V2.
Image Caching and delivery issues
Furthermore, we observed that photos were loading slower for some reason, so we began investigating it from our servers to our global CDN provider, which caches images all over the world. It was discovered that it was not the fault of our CDN provider, but that we were caching images on our servers many times, causing the request to take 5 to 6 seconds to fully complete with all image optimizations. Also, when sending images and files to our CDN provider, we optimized the request and deleted session data and other items that were unnecessary.
We are continually striving to improve and learning from our mistakes. With all of these optimizations and efforts to strengthen our network to better serve our users, last month was a big success.