Frequently Asked Questions
We are truly sorry for the technical issues that have impacted you all since January the 5th.
Tapestry has been stable since Tuesday the 12th of January, although some of you might have experienced delays with notifications, scheduled observations and antivirus scans.
We understand how frustrating this has been for all of you struggling to provide remote learning for your children. We are working our absolute hardest to get to the bottom of the issue.
We anticipate a surge of activity again on Monday after the weekend, and we have been making preparations for this. Our developers continue to put all their efforts into monitoring our database.
We will continue to update our FAQ page regularly with any news and changes that have been made. If there is anything you would like to know, but that isn’t covered on this page, please email customer.service@eyfs.info.
Page last updated: 16:15 GMT Friday 15th.
What are the symptoms of the issue?
As you move around and use Tapestry, you may be taken to error pages. In some cases that is a 500 error page (a drawing of two dogs, a server, and one of our devs), and in some cases, it’s a 504 error (a text page). Either way, whatever you were trying to do will not have worked and you will need to try again.
It is affecting some Tapestry accounts more severely than others. Most people will be experiencing errors to an extent, but some of you will be experiencing a lot of them. The error rates are also not consistent within accounts, so you may experience quite a lot of them in a short burst, and then no more for a while.
To help reduce these we are turning off ‘background processes’ as soon as we can see load on our servers is high. What this means in terms of symptoms is:
– Notifications will stop going out.
– Any media you upload will successfully attach, provided you don’t see an error page, but it will not be ‘processed’ and therefore won’t be visible.
– Documents you upload will successfully attach, provided you don’t see an error page, but the virus check we run on them will not start and therefore they will not be downloadable.
– PDFs you set to export will not become available to download.
– Scheduled observations will not be published.
These things are not lost, they are in a queue and will go out slowly at points when our servers are stable – usually overnight.
Is high traffic causing this?
Rather than being the cause, high traffic is revealing a deeper issue that our developers are investigating. The exceptional level of activity is causing a chain reaction on our database servers. Our technical summary gives a more detailed explanation of this.
What is causing the problem?
Whilst the issue is occurring because of very high traffic, it is not one of scalability, but rather a complicated technical problem we are trying to resolve. Those with technical expertise may want to read our technical explanation.
What are you doing to fix it?
Each day, as soon as we can see the servers starting to struggle, we are turning off what we call ‘background processes’. This reduces the strain on our servers and means you’re more likely to be able to move around Tapestry, view existing posts, and add new posts successfully. However, the benefits of this will only be seen by some users, and it is not a long term solution.
You will be able to see whether the background processes are on or off from the Tapestry Status page.
We have increased our database servers, where the issue lies, to the largest ones that our hosting company, AWS, can provide.
We have also set systems up to closely watch exactly what is happening to our databases when these errors start – this is known as logging. That should give us a more detailed insight into the problem.
On Monday night we made a background change to Tapestry to share the load better across our database servers to try and mitigate the problem whilst we work on investigating and fixing the underlying cause.
Our developers are putting every effort into diagnosing and solving the issue.
What improvements can we see?
Tapestry has remained stable for most of the past week, following some changes made on the evening of Monday 11th January. We have not experienced any errors since then although we have seen a delay in some notifications being sent, some scheduled obs being published, and some documents getting through the antivirus scan.
However, the core issue remains and our developers are putting all their efforts into resolving this with the aid of AWS – the company that hosts our servers.
When will it be fixed?
How will you keep me informed?
We will keep this page updated and announce when we have done so on our Twitter and Tapestry Status page.