After upgrading to 7.2.1 and rebooting, I have no tags from the default provider. My external DB driven ones are still there, just the internal one. I can create a new internal provider and create new tags there, but I’d really, really, like to get my old ones back.
Noe please, I don’t need to hear the “Did you make a backup” question, because no, I didn’t.
Also, I don’t need any post about “save early, save often” because nominally, I do. We’re all imperfect humans in an imperfect word. OK, I’m done soapboxing.
The point of the post is: can I repair the existing installation? It seems almost more of a pointer kind issue, than a fatal error…
Now that I’ve shot myself in the foot, where do I start hunting?
I’d quickly go grab the files in the “autobackup” directory under “{Install Dir}\contexts\main\db\autobackup” and copy them aside. Upload the oldest and most recent to the uploads area using ticket id #1887.
You should probably also shut down the gateway and zip up the main files in that “db” directory (just the various “settings” files) and upload that as well.
Hopefully it’s something easy enough to recover from.
The only way I could clear up the context issue was with a reinstall. Which means I’ll have to call in to reset the key, but a small price to pay. It at least got me back to where I started this morning!
Had the same problem with last Friday’s autobackup, but Thursday’s looks to be okay and I have my tags back. I’d say we can close the support ticket. Do you still want the two backups to compare?
Oh, great! No, if you got it working, I don’t suppose there’s too much to look at. I’d recommend making backups though
Along those lines, if you’re not aware, there’s a handy “Scheduled Backup” tool in the gateway under “Configuration>Gateway Settings”. If you turn that on it will create a backup by default every day, in addition to the autobackups made by the db system. You can point it to a network file share and never have to worry again.
I think i figured out root cause on this issue. When our illustrious IT Department got me this server box to play with, they partitioned out 20Gb for the system (C:). This is also where Ignition got installed. Couple that with some very large log files from a bad OPC connection, and whamo! Out of disk space.
Did you know that running out of disk space can cause some interesting side effects to Ignition? It’s also amazing how quickly one’s vocabulary grows as one is cursing at the system…
Much quicker recovery, now that I know what to look for, but I thought I’d give a heads up.
Ah ha, thanks for posting back with the cause. Yes - running out of disk space causes all sorts of unpredictable havoc on software (especially running on Windows).
We have a vague notion of adding a bunch of Gateway-scoped system tags / alerts that would warn you of certain dangerous conditions before they became catastrophic (I know, what a novel idea, huh?)
We’d thought of some obvious things like running low on memory, detectable deadlocks, database connection pools filling up and prolonged high CPU usage. I’ll add low disk space in installed drive to the list - thats a good one.
Was it Ignition logs that grew that large? Were you running a dev build or any special debug settings? I thought I remember one of the Ignition updates addressing this issue. 20 gigs of log files seems like a bad thing. Carl or Colby - any advice here?
Ignition logs grew to about 8GB over the course of a few of days (one entry for every bad OPC write a second adds up), the virtual disk space was on here, and with all the support software (mostly MySQL core, MSOffice and a PDF batch traslator), it just got to be too much for the tiny partition and it cracked under the strain.
I’ve move the virtual disk to another partition, and when time permits, I’ll resize the system partition to something reasonable-- like add another 20GB
If you are running on a system with limited disk space, there are a couple of settings you should set in the ignition.conf file.
# Maximum size that the log file will be allowed to grow to before
# the log is rolled. Size is specified in bytes. The default value
# of 0, disables log rolling. May abbreviate with the 'k' (kb) or
# 'm' (mb) suffix. For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=0
# Maximum number of rolled log files which will be allowed before old
# files are deleted. The default value of 0 implies no limit.
wrapper.logfile.maxfiles=0
Unfortunately the default is to create one log file and let it grow until there is no room left on the disk.
Carl, Colby
Perhaps the defaults could be changed to only consume say, 5G? in 5 1G files?
Unfortunately its not just the wrapper.log files, its also the Ignition\contexts\main\logs folder. When I remember I make sure that it will only keep 5 10 MB wrapper files but the bad ones are the HTML log files. I’ve had 8 GB HTML files in a single day and on my server it only has 20 GB of space which can hurt.
I’ve setup an automated script which gzip compresses the HTML files once a day after they rotate (on top of making the folder real-time compressed) and then prunes the directory periodically. While its possible to tune the log4j verbosity so repeating messages are removed usually I find about it after its too late.
For reference we ran out space after our last upgrade because I forgot to reset the wrapper.conf file to only keep 5 files after the install. I recovered most of it from an automated backup but Ignition\tomcat\webapps\ROOT\WEB-INF\web.xml was already corrupted and not recoverable. I grabbed a copy from another machine and restored it manually. Now I keep a second copy of just around just in case it happens again.
