Archive for the ‘Operations’ Category
An apology, an explanation and a commitment
In the last 30 days Yola has had far too many incidents of downtime. The engineers at Yola who create the systems and take responsibility for them being up and available find this unacceptable.
Let me highlight the area where we performed worst in terms of uptime – the Published Sites. This means your site that you have created and host with us. In the past 30 days, if you have a live site, it has been unavailable for a total of 3 hours. The longest time it was unavailable was 43 minutes, and the shortest time was just one minute.
As engineers, we take the uptime of your site very seriously, even personally. We are extremely proud when we get to the end of a month and there has been no downtime. It is usually the area where we have the least downtime – because we put an extraordinary amount of effort into keeping your site up and fast. We know what it means when your site is unavailable – it could be at a time critical to your business or personal brand and you could be losing money. When your site is unavailable you feel helpless and frustrated.
As the engineer who takes responsibility for everyone involved in keeping your site up, I would like to apologize to you, explain why it has been such a rough month and tell you what we are doing to make sure it never happens again.
There has been no disaster in our service infrastructure, as you might expect after such a bad month. In fact most of the issues have been unpredicted (and honestly, completely unlikely) side effects of some real, meaningful upgrades to the infrastructure that runs our millions of hosted sites. There is always a very small chance that no matter how well you plan for upgrades to happen ‘transparently’ and professionally, something could go wrong. It feels like we have have been quite unlucky in this regard.
The upgrades we have been doing involve a process of removing our reliance on infrastructure that is known to fail sometimes, so that we are left with infrastructure that can cope when things go wrong and does not cause downtime. But more than once in the last month, that failure-prone infrastructure would fail as we were removing it, causing downtime. For those of you interested in detail, the issues have included operating system failures and network filesystem failures.
Here is a commitment to you.
- Your site’s availability and uptime is our number one priority.
- Your site data is safe and secure. We take care of backing it up and making sure we can recover it in the event of things like natural disasters.
- We will continue to improve your site’s uptime and response time.
- We strive for 100% uptime on your site.
- The same applies to our Site Builder and MyYola, although the uptime of your site will always take precedence over these if we need to make that call.
Please contact me if you have any questions whatsoever.
Lisa Retief
VP of Engineering at Yola
Labels: Operations,Service,Upgrades | 12 Comments »
Recent Yola outage: Here’s what happened
Yesterday was a very difficult day for Yola and our customers. Yola’s servers were attacked unexpectedly and maliciously by as yet unknown individuals and/or groups, bringing down all of our hosted websites as well as our service provider’s entire datacenter.
The thing Yola cares about most is providing a world class service to all our customers, and doing so consistently and reliably. When our servers are attacked, your website is attacked. We don’t take this lightly. We worked around the clock to bring the service back up, and talked to you in our customer community forum, Twitter, Facebook and email during the outage to answer your questions and keep you up-to-date.
We’re posting this information on our blog now to tell you what happened, what we’re going to do about it in the future, and, most importantly, to apologize. Regardless of the reasons, your website was down. Please know that however frustrated we were with the situation, we understand you were more so. We’re very sorry.
Here’s what happened and what we plan to do about it:
Labels: Operations | 56 Comments »
New Features Live
We’re back up and have a very full and exciting release for you: Blogging, domain registration and shopping cart widgets! Check out the details below:
1.) Blogging
The new blogging feature allows you to post online journal entries, text, images and share it with the world within minutes.
Features include:
- Instantly create a blog post that is dated, categorized, includes tags and author details
- Blog manager to search and access all your posts within a few simple clicks
- RSS syndication
- Commenting
- Create multiple blogs under one site
To add a blog to your existing site, simply click on the “New Page” button. To create a new blog, click on “Create a New Site”.
2.) Domain Registration
Buy an available domain (www.yourdomain.com) with a few simple clicks! Take the hassle out of configuring your new domain by buying it through us.
You’ll see this option when you first publish your new site. If your site’s already been published, you can use this feature by unpublishing your site first and then republishing.
3.) Shopping cart widgets
We’ve integrated with PayPal and now you can:
- Sell anything on your website and securely collect payments from your visitors via PayPal
- Accept credit card and bank account donations from a button on your website
To add any of our new Paypal widgets, click on “Online Store” in the content sidebar in the site builder on the right hand side of your screen.
Try these features out and tell us what you think!
Labels: Operations,Releases | 52 Comments »
We’re adding new features to SynthaSite
We’re releasing a bunch of new features today and the update is taking a bit longer than we expected. We’ve had so many new users join us over the past couple of months that its taking a while to update everything.
In the meantime, your published sites are available and we expect to have the site builder available for login soon. Thanks for your patience. You’ll love the new features we’re releasing.
Labels: Operations,Releases,Upgrades | 38 Comments »
Service Restored
Service has now been restored to the hosting environment and all published sites should now respond as usual.
An unusual combination of events caused a temporary networking hiccup to become a bigger problem that affected our hosting environment. We’ve identified a way to avoid this type of failure in future, and will be implementing it shortly.
We apologize for the inconvenience and appreciate your support as we work to improve our service.
Labels: Operations | 8 Comments »
Service Interruption
We are currently experiencing technical difficulties. Users may get an error when trying to access published sites. We are investigating the problem and hope to have the service restored as quickly as possible.
We apologize for this service interruption and will post updates here as soon as the problem is resolved.
Labels: Operations | 1 Comment »
Update on last night’s downtime
At around 21h00 PST on Wednesday night 30 July, our Site Builder
service became unavailable to users. This was due to a hard disk
failure on a critical server. There was a backup drive waiting to take
over, but for a reason still unknown to us, it did not happen. The
Site Builder (the service you log in to in order to create, edit and
publish sites) was down for about two hours while we diagnosed the
problem and switched over to our failover systems. The published sites
were never unavailable so there was no disruption for the users of
your sites.
Hardware failure is something we know will happen from time to time,
no matter how many preventative measures are put in place. So as
engineers we plan and rehearse for it so that we are able to respond
with least disruption to our user base. In this instance the longest
part of the downtime was spent making sure we knew exactly what had
gone wrong so that we could make the correct decisions for recovery
without compromising the integrity of any data.
I want to extend an apology to all users who were unable to log in
during this time. During such incidents you are foremost in our minds
while we work to bring everything online again.
SynthaSite is uncompromising in our commitment to optimal availability
and reliability of the platforms that run both the Site Builder and
the sites you host with us. We have asked the data centre where the
server is hosted to investigate the reason for it not automatically
failing over to another drive, and will take steps to prevent this
from happening in the future.
Labels: Operations,Service | 4 Comments »
Improvements to SynthaSite and some downtime
We’re making some exciting changes to our user interface and need to take down the site builder this Thursday (July 10, 2008) between 1am and 4am PST. During this time, you will not be able to make changes or updates to your sites. Your published sites will stay up and your visitors will still be able to visit your sites.
Thanks for your patience. We think you’ll really like the changes we’re making!
Labels: Operations,Releases,Upgrades | 10 Comments »
Planned maintenance
SynthaSite is upgrading a server component on Friday (20 June 2008) morning and need to take the SynthaSite Site Builder offline between 1am and 2am PST. During this time, you will not be able to log in and make changes or updates to your sites. Your published sites will be unaffected.
Labels: Operations,Service,Upgrades | 6 Comments »
Maintenance completed
We have completed the scheduled maintenance and the SynthaSite services are back in working order. It took a little longer that initially anticipated – thank you for your patience.
As promised there was no downtime on the published (hosted) sites.
We are aware of a known issue which occurred as a result of the changes we made – image editing from within the Site Builder will currently not work. This is being worked on and we will let you know when it is resolved.
Labels: Operations,Service | 7 Comments »