01 Dec 2014
The following post is a guest post by Eoin O'Driscoll (web developer), and Vinny Glennon (co-founder) of FanFootage.com
When we founded FanFootage we knew there was something lacking in the concert
and event experience. Fans were taking videos on their mobile phones during
performances, loading them on YouTube, and the audio was horrible. So we
created FanFootage where fans can upload their own video and we work with the
bands to get the high def audio and put them together. Now fans can come to
our site and see their favorite concerts or sports from any angle. They can
also search for upcoming events and performances.
For a recent Linkin Park concert the band and the fan group contacted us.
Their fans uploaded more than 1,500 videos from almost every angle. On
FanFootage that single concert has had over 350,000 page views.
Our user experience heavily relies on search
Earlier in our design we knew that search would be key to making FanFootage
the ultimate fan experience. When a user comes to our site the first thing
they do is search for the event or artist. And we need to make sure that they
either find the artist they are looking for or something similar, and it has
to be fast.
As developers, our team isn't new to search, particularly within the
entertainment space. Our previous startup in the music space was bought by
RealNetworks and a second startup was a competitor to Google. That is where we
learned that search is hard. And when we thought to build our own search on
FanFootage we quickly said it wasn't going to happen.
We also know what fans need. User demands have changed now that they can
access anything from their phones. Today we expect our applications and
services to predict what we are going to do next. And because of Google,
people don't search with a single phrase. Users expect search to understand
how phrases fit together and are related and of course it needs to spell check
and it must be instant.
We also had different search requirements than other sites. Normally search on
a site is for one unit or concept; a site for flowers for example. For us we
needed to allow fans to search for artists, bands, friends or upcoming events
in their area and never get a zero result.
Why we chose Algolia
After looking at a few search applications we agreed on Algolia.Many search
applications look nice but don't have the flexibility we needed to configure
them they way our business needed. And most weren't fast.
Why did we chose Algolia? First it has a developer-centric approach. It
took us 2 hours to configure and a day to test and that was it. We basically
had search up and running in a day. The dashboard lets me know that the API
calls are returned within milliseconds and we have all the flexibility we need
to configure as our content grows.
Today, more than 250 artists have used FanFootage in 20 countries. We are
growing quickly. As a company we are still learning what our fans are
searching for and Algolia is helping us with that. As content grows we will
continue to configure search to meet the needs of our fans. We will also be
rolling out Algolia for mobile because of its multi-search capabilities.
Algolia is a simple solution to a complex problem. And it blew our mind away.
It just works. And now we can focus on our own fanbase.
Images courtesy of FanFootage. Learn more on their
website
28 Nov 2014
Today we had a severe DNS issue that impacted some of our users during a total
of 5 hours. Although most of our customers were not impacted, for some of them
search simply went down. This event and its details deserved to be fully
disclosed.
The context
Up until recently, we were using Amazon Route 53 for our DNS routing needs.
When we started to design our Distributed Search Network
(algolia.com/dsn a few months ago, we quickly
realized that our needs were out of Route 53's scope: we needed a custom
routing per user and the two options of Route 53 simply didn't work:
- Latency-based routing is limited to the 9 regions of AWS and we have 12;
- With geography-based routing you need to indicate country per country how you want to resolve the IP.
This is a tedious process for a not even good solution as route 53 does not
support EDNS right now.
So we started to look for new DNS options. Choosing the best DNS provider is
not something you do overnight. It took us months to benchmark several vendors
and find the right one: NSOne. The filter chain feature of
NSOne was a
perfect fit for our use case and the NSOne team was great in understanding our
needs and even went the extra mile for us by building a specific module,
allowing better performance.
Something we also discovered during this benchmark was that the algolia.io
domain was not good for performance compared to algolia.net, as there are far
more DNS servers in the .net anycast network than in the .io one. The NSOne
team offered us a smart solution based on linked domain, so we wouldn't have
to maintain two zones ourselves.
The migration
The goal of the migration was to move from Route 53 to NSOne. For several
weeks we have been working on importing the records in NSOne and making sure
Route 53 and NSOne were synchronized. Our initial tests revealed some issues
but after a few days of continuous updates without any difference between
Route 53 and NSOne, we started to be confident about our synchronization and
started the migration of the demos of our
website to make them target the new
algolia.net domain. We tested the performance and resolution from all NSOne
POP (https://nsone.net/technology/network/) to be sure there were no glitches.
These first production tests were successful, synchronization was ok,
performance and routing were good, so we decided to move the .io domain from
Route 53 to NSOne as well.
The D-day
The big issue when changing the DNS is that it is global and involves caching
logics, making rollbacking complex. With users in 45 countries it is almost
impossible to find a suitable time for everyone: DNS changes cannot be done
gradually. We decided to push the update during the night for the US, at 4am
EST.
We witnessed a quickly rising number of queries targeting NSOne and it's once
we reached about 1,000 DNS queries per second that we started to receive our
first complain about failed DNS resolution. This routing issue was not
impacting all DNS resolutions but some of them were replying with a NXDOMAIN
answer, the equivalent of a DNS "404 not found".
$ dig APPID-1.algolia.io`
; <<>> DiG 9.9.5-4.3-Ubuntu <<>> APPID-1.algolia.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER< ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;APPID-1.algolia.io. IN A
;; AUTHORITY SECTION:
algolia.io. 45 IN SOA dns1.p03.nsone.net. hostmaster.nsone.net.
1414873854 43200 7200 1209600 3600
;; Query time: 24 msec
;; SERVER: 213.133.100.100#53(213.133.100.100)
;; WHEN: Thu Nov 27 10:49:33 CET 2014
;; MSG SIZE rcvd: 115
After double-checking our DNS zone for those specific records, we understood
it was a NSOne bug related to our custom routing code. We immediately
rollbacked to Route 53.
The NSOne support was really quick to react and they found the issue pretty
quickly: the issue only concerned some DNS with EDNS support on the algolia.io
domain. The algolia.net domain was not impacted, explaining why all the tests
we've done weren't able to detect the issue before.
Unfortunately, it did not stop here and something very unexpected happened:
some customers (even not priorly impacted) started to face issues right after
the rollback to Route 53.
In order to improve performance, the custom Algolia module developed by NSOne
was doing some translation on our records: APPID-1.algolia.io is translated
into 1.APPID.algolia.io and then resolved to CNAME for the actual server in
the cluster serving that customer. The translation of APPID-1.algolia.io to
1.APPID.algolia.io was done with a TTL of 86400 seconds (1 day). Since these
zones did not exist in Route 53 before, it was not possible to resolve there
records anymore. What made the situation even worse was the TTL far exceeding
the TTL of NS records. Most of the DNS servers flushed their cache for the
domain, once the nameservers changed. But the remaining ones kept the record
cached.
TL;DR: Do not forget about IPv6. As if it was not enough, we eventually discovered something else: our custom DNS module was resolving APPID-X.algolia.io to X.APPID.algolia.io only in a case that there were no direct resolutions to an IP address. This translation worked pretty well as we had all the A records set. But some customers started to report weird resolutions. Normally we resolve APPID-1.algolia.io -> 1.APPID.algolia.io -> servername-1.algolia.io -> IP. Which was completely fine until the moment an IPv6 AAAA request came. Since we did not have AAAA records, the custom filter started to resolve: APPID-1.algolia.io -> 1. APPID.algolia.io -> servername-1.algolia.io -> 1.servername.algolia.io -> nothing.
We were in a bad situation feared by all engineers, this lonely moment when
you really miss a "purge cache" feature.
Eventually, as soon as we got confirmation of the fix by NSOne, we changed
again the DNS of algolia.io to NSOne and helped our customers to workaround
the issue before the cache expiration:
for our customers impacted by the NXDOMAIN issue, a simple migration to the
algolia.net domain instead of the algolia.io problem fixed the issue;
for those impacted by the Route 53 rollback issue, we created new DNS records
for them to avoid work-around DNS caches.
Conclusion: what we learned
This is by far the biggest problem we have encountered since the launch of
Algolia. Although the first issue was almost impossible to anticipate, we have
made mistakes and should have handled a few things differently:
DNS is a highly critical component and being the first to use an external
custom module was not a good idea, even if it improved performance;
Putting more thought into the rollback part of our deployment would have
helped us anticipate the second issue. For a component as critical as a DNS,
having a robust rollback process is mandatory, no matter how much work it
represents and even though such an event is extremely unlikely to happen.
We're very sorry for this disruption. We wanted to share these technical
details to shed some light on what happened and what we've done in response.
Thanks for your patience and support.
If you think we missed anything or if you'd like to share your advice on your
own best practices, your comments are really welcome.
14 Nov 2014
Algolia Increases Online Search Sessions By 60% and Unique Mobile Searches
by 270%
The following post is a guest post by Omar Kassim, co-founder of JadoPado.
Founded in 2010, JadoPado is one of the largest e-commerce
sites servicing the GCC, Middle East, North Africa and South Asia. Its CEO
Omar Kassim wanted to bring an Amazon-like experience to the region. In just
3 years of operations the company now boasts thousands of customers, hundreds
of vendors and over $7 million in annual revenues.
Realizing that search is a key component of their user experience and engagement, Omar and his small team of 15 set off to build new search capabilities that would help users find the products they wanted, lightning fast. In addition, the team was developing a revamped mobile experience and saw that search needed to be spot on for both smartphones and tablets. "I saw search as a competitive tool and as a strategy to get a leg up on our competition. After seeing Algolia on Hacker News I was absolutely blown away. After looking at the demos, we threw out what we were doing internally in terms of a small search revamp and I had one of our team get cracking with Algolia right away. As a little startup, it really helped that Algolia's price points were within reach in terms of not breaking the bank to get things rolling."
The Power of Instant Search
After configuring and testing Algolia for two weeks, JadoPado had the results
they were looking for. Branded internally as InstaSearch, JadoPado knew that
it would dramatically improve how search functioned on both mobile and the web
at JadoPado. "The idea from the outset was to build InstaSearch. I kept
ending up at the Algolia demo and thought it would be incredible if we could
forget all user interaction aside from typing and just display results right
away. Remove what you've typed and the results disappear taking you back to
where you were. We then spent a bit of time figuring out how to get each
result "page" to have a URL that could be used with external search or shared
elsewhere," explained Omar.
Making Search Seamless
"We looked at a number of solutions. One of our biggest intentions was to
try to get search to be extremely fast and as slick as possible. Customers
should feel like search "just works" and that it is a super easy way to get
straight to to whatever they may be looking for. Algolia has allowed us to
accomplish that," Omar explained. "Moving search from a not really
working internal model to a search as a service platform has allowed us to
focus on other areas while knowing that search works and that we've got an
edge over our competition."
Support For Arabic
With more than 20 countries to support, the JadoPado team knew that the key to
success in the region was to ensure that search be delivered in Arabic as
well. Omar explained, "The final bits were figuring out a separate set of
indexes for Arabic (as we were about to roll out a standalone Arabic version
of JadoPado) and getting the faceting right. This was easy to do with the deep
Algolia documentation." Algolia works with all languages, including
Chinese, Japanese, Korean, and Arabic. No specific configuration required,
speed and ranking perform exactly the same way.
Better Business Through Search
In May the team rolled out InstaSearch, Arabic support and a newly revamped
mobile experience with search at the center. JadoPado immediately experienced
a doubling in conversions and activity that was triple a typical day.
Compared to the same 30 day period in 2013, JadoPado saw an increase in site
visits through search from 8.2% to 11.3%.
Additionally:
- Sessions with search has jumped 59.96%.
- Unique searches has jumped 46.87%
- Average search depth has increased by 58.87%.
Mastering Mobile Through Search
The greater impact of Algolia's hosted search was JadoPado's revamped mobile experience. Search is often the first action customers take on a mobile device. With instant search, autocorrect and full language support, improving search and the quality of results can have a significant impact on revenues. With Algolia implemented as part of JadoPado's mobile site, the company saw strong results with visits from search increasing from 4.3% to 15% over the same time period and session exits decreasing by 16.57%. A big change. And search increased engagement on all levels:
- Mobile sessions with search jumped by 233.92%
- Total unique mobile searches jumped 268.37%
- Average search depth on mobile devices jumped by 41.05%.
Images courtesy of JadoPado. Learn more on their
website.
10 Oct 2014
Algolia Speeds Up Search Result Delivery Times From 10 Seconds To 250
Milliseconds.The following post is a guest post by Teddy
Chan, Founder and CEO at
AfterShip.
AfterShip is an online tracking platform which helps online merchants track
their shipment across multiple carriers and notify their customers via email
or mobile. Being an online merchant myself, I shipped more than 30,000
packages a month around the world. When customers contacted me to get an
update on shipments I realized that I couldn't track shipments from different
carriers and get updates on their status in a single place. So I built
Aftership to allow both consumers and online merchants view all their packages
on a single platform.
After winning the 2011 Global Startup Battle and 2011 Startup Weekend Hong
Kong Aftership opened into beta and quickly helped thousands of online
merchants to send out over 1,000,000 notifications to customers.
One of the key parts of our service is providing customers around the world
with up-to-date information about their packages.
Right now we have more than 10 million tracking numbers in our database. This
causes a few different challenges when it comes to search and we needed
technology that would help us continuously index constantly changing
information.
Our first challenge is that we are a small team with only 1 engineer.
We are not in the search business, so we needed a solution that would be easy
to implement and work well with our existing infrastructure. Algolia's
extensive documentation made it easy to see that our set up and implementation
time would be extremely fast and would work with any language and database, so
we could get back to our core business.
Algolia was super easy, we had it tested, up and running in a week.
Our second challenge was quickly delivering search results.
On Redis, searching for packages was simply impossible. For each query, it
would simply lock up until the result was found, so it could run only one
search at a time. Each search with Redis was taking up to 10 seconds. With
Algolia we reduced search result delivery times to 250 milliseconds for any
customer anywhere in the world. When you think about thousands of merchants
who send more than 1 million packages per month, you can see how speed is
critical.
Downtime also is not an option when tracking packages around the globe.
We are very strict when adopting new technologies and SaaS technologies can't
slow down our system.
Algolia had the highest uptime of the other solutions we looked at. There was no physical downtime.
Our final challenge was search complexity.
Sometimes you need to know how many shipments are coming from Hong Kong and
exactly where they are in transit to and from the U.S.. Shipments going around
the globe can change status several times within a single day. With Algolia's
indexing we are able to instantly deliver up-to-date notifications on all 10
million packages, so that customers can not only track their package on its
journey, but they can also go to their online merchant's shop and see a real-
time status of their package.
In the end, it was Algolia's customer service that won us over.
Similar services and platforms were not responsive. With Algolia we either had
the documentation we needed, immediately were able to get advice from an
engineer or had our problem solved in less than a day. With such a small team
this means a lot. And with the Enterprise package we know that Algolia will
grow with us as quickly as our business does.
Want to find out more about the Algolia experience ?
Discover and try it here
01 Sep 2014
Do you know the 3 most important things in search? Speed, speed, and speed!
At Algolia, we work at making access to content and information completely
seamless. And that can only be done if search results are returned so fast
that they seem instant.
That means two things for us: getting server response time under 10ms
(checked), and getting the servers close to end-users to lower latency.
We are on a quest to make search faster than 100ms from anywhere in the world,
and today is an important step. We are thrilled to announce the opening of
our 4th datacenter, located in California!
You can now choose to be hosted on this datacenter when signing
up (multi-datacenter distribution is
also available for enterprise users).