Truth in IT
    • Sign In
    • Register
        • Videos
        • Channels
        • Pages
        • Galleries
        • News
        • Events
        • All
Truth in IT Truth in IT
  • Data Management ▼
    • Converged Infrastructure
    • DevOps
    • Networking
    • Storage
    • Virtualization
  • Cybersecurity ▼
    • Application Security
    • Backup & Recovery
    • Data Security
    • Identity & Access Management (IAM)
    • Zero Trust
    • Compliance & GRC
    • Endpoint Security
  • Cloud ▼
    • Hybrid Cloud
    • Private Cloud
    • Public Cloud
  • Webinar Library
  • TiPs
  • DRAW

Veeam: Beyond Immutability: Backup Testing & Forensics Best Practices

Veeam
09/05/2026
0 (0%)
Share
  • Comments
  • Download
  • Transcript
Report Like Favorite
  • Share/Embed
  • Email
Link
Embed

Transcript


I'm back from vacation. I was like two weeks out and now Rick, as you can see, is not joining us today. He's at an important event and couldn't make it. But as always, we actually find ways to keep things going. I have two special guests and as well, Sophia that is not anymore. She's more like a regular. Not that special because you are more like a regular. You always cover for me with Rick. Welcome all and thank you for taking the time to do this with me today. For the ones that are watching us for the first time, maybe it's worth going through some introductions. I'm going to start with you, Jonah. Would you like to just introduce yourself? Sure. Thanks, Maddie. Hi, everyone. If you haven't met me, I'm Jonah May. I am a Veeam Vanguard. I am a user group leader for the automation desk and for Texas. I'm an object-first ace. I'm Veeam certified and I work at a partner that's a Cloud and service provider called Cyber Fortress. Maurice. Hi. My name is Maurice. I live in the Netherlands. I am a Vanguard and object-first ace as well. Automation desk and for the VOGNL, the leader. Aside of that, both Jonah and I are co-founders of the Veeam Community Hackathon. Awesome. Good stuff. Guys, if you didn't know what are we going to talk about, you can see a lot of background with Hackathon and my T-shirt, because I decided not to have a background, but the T-shirt today just to be special as well. We're going to have a bit of fun. We're going to go through some articles, of course, as always. We picked three from the community members, so that's going to be fun to discuss. Great topics as always. Then we're going to talk a little bit about the Hackathon and of course, special department news. A lot of great content from our community members. We selected three and we're going to start with Ken's one. That was a really interesting topic in here. Ken talks about why immutable backups are great, but they are not enough on their own. I thought this was a great read because everyone talks about immutability as the silver bullet, but then Ken points out it's really just the first layer when we talk protection. Actually, I'm going to talk less about the topic and I want to ask both Jonah and Maurice, what do they think about this topic? How do they see it? If they have ever had a situation where backups were technically there, but recovery failed, do you want to start, Maurice, with that one? I haven't had an issue that backups were available, but failed luckily enough. But yeah, you need to be sure that your backups are actually working and testing that is a really, really, really important part of the whole backup strategy that you need to create in your environment. Because if you don't and you need your backup, and you don't have your data, yeah, well, we can all think of what will happen then. Exactly. Then you fail, right? So testing is really important, actually. There are great, great possibilities and things that you can do to automate those testings within Veeam as of today already. Including but not limited to with Veeam Recovery Orchestrator. Absolutely. Yeah, absolutely. That's very correct. What do you think, Jonah? I agree. Testing is important, but immutability is just one of what should be multiple layers of security. I have seen corrupted backups before where they're still there in multiple methods, including it's not very common, but I've seen multiple instances out there in the wild where a customer actually has immutable backups with an object storage vendor and the data is all still there, but the bad actor has essentially managed to compromise the Windows Server that Veeam was running on extract the S3 keys. Then for those of you who aren't super familiar with object storage, part of how it's made possible is through what they call file versioning. The bad actor actually went in and messed around with the file versions, added a whole bunch of new versions that were full of garbage data, and it caused all synchronization issues where the data was still there, the data was still accessible, but you would have essentially had to undo new file versions on millions of objects, which takes a very long time. They actually couldn't use their data in their S3 bucket because while it was there, it wasn't accessible. The reason that that customer was saved was through other copies of their data, hardened Linux repositories or Cloud Connect backups. But it's not even just multiple copies of the data. Immutability is only at the storage layer. There's still other layers that social engineering can occur or hardware failures can occur. Resiliency is important and security is important. Things like multi-user authentication, multi-factor authentication, redundant copies of your data, all of it's important to build a comprehensive strategy. At my company, we not only do backup, but we also offer security services. We say we have what we call the Trinity platform or our three pillars, which is not only recover, but also prevent and detect. That's important. It's all important to your comprehensive strategy. You need to have multiple options. Your backups need backups almost. If plan A fails, what's your plan B? What's your plan C? Exactly. Actually, my next question was, how do you make sure your daily backup operations don't get neglected? Because everyone gets busy. Ken has a lot of great points in here, but you mentioned Jonah, and you mentioned MFA and all these ways of making sure your data is properly backed up and tested, and you use the right tools for it. But let me ask as well, Maurice, on that side of things, how do you make sure, for instance, at your company that the backup operations, they don't get neglected? If you have any advice for our audience, I mean, from both of you, that would be great. You are both technical experts. You are really good at what you're doing, so I'm pretty sure the audience is going to appreciate any good advice. Yeah. Well, keep in mind that every piece of software will have a vulnerability at one point, so keep it up-to-date. Don't slack with that. Just continue updating, including your operating systems, because that should be part of your strategy as well, keeping up-to-date. I don't say late assertion. If you're running version 12 for reasons, that could be okay as long as it's supported. But if you're still on version 11, then I would say update yesterday. So yeah, keep up-to-date with all the patches. Keep up-to-date with all the strategies that are out there, the advices that people like can hear in this post, create, and keep on the money, basically. Yeah. That's pretty good in there, advice. Jonah, what do you think about it? I mean, you mentioned it. Do you think anything in Ken's blog is more important than? The rest of the information he shared with us, anything that stands out for you from what he shared? I think the recovery testing is really the most important. I mean, clean restore points, you can piggyback off your recovery testing. You know, as part of your recovery testing, you can validate that your data is clean. And then a lot of it is good. A lot of it is even being best practices and things I've recommended for years. But I think it's very important, and I'm very glad that he put recovery testing as number one because I like what he says. Testing proves you can use it. And if you haven't tested restores, you're still guessing because something I've always said in webinars and on calls with potential customers in sales engineering is, you know, if you haven't tested your backups, you don't have backups. Yeah, that's a good one, actually. Yeah, I like it. Another thing that I would like to add to this story is make a plan for if you don't, if you're not able to restore to your own hardware. What if you get ransomware, Mr. FBI or your local police department comes around and puts this white, yellow, white, red ribbon lint around your servers? What's next? How can you recover? How can you get up and running again? What can you do? Exactly. Yeah, that's a very good point, Maryse, because I've seen that a number of times on the cloud and service provider side. You know, we obviously, and you probably do too, also working at a service provider, offer that off-site recovery location. And we 100% have seen where insurance companies or federal agencies have come in and essentially said, hey, you're not allowed to touch production. We need a full root cause analysis and to figure out how the penetration occurred, when the penetration occurred, how it spread. Forensic analysis. That's the word I was looking for. Yeah, and we're actually going to talk. I think our third article is on that. So good stuff in here. Yeah, thank you for that. And thank you, Ken, very much for this great article. Keep on being a regular. You became a regular. So great content that you are sharing with us. Just keep the great work. And let's move on the next article. This one is from Andreas. He works for Veeam. And I thought this was also like an interesting topic, talking about adding SC hosts to Veeam. Specifically, he's talking whether it's OK to add standalone hosts or if it's better to go through vCenter. And what he's doing here in this article, he points out a few things that are important. He says, without vCenter, Veeam can lose track of VMs that move between hosts. And then dynamic job management with tags and folder is limited. And then managing credentials across multiple hosts becomes more weak. I'm going to just ask the both of you, what do you think about this article and if you have any experience with it? Yeah, I have added VMware hosts without a vCenter to Veeam. And the only reason I would like to do that is in case I need to restore a vCenter. Because that's, in my opinion, the only reason why you should, if you have a vCenter and your VMs can move between ESXi hosts, it's the only reason that you want to have a standalone host added, unless you don't have a vCenter, obviously. The vCenter, especially with DRS and stuff, will move VMs over from host A to host B without notice. Yeah. I think it's a great topic because it is a conversation that I've had with customers before as they're looking to build or rebuild their Veeam environments. Especially if it's a company where they have a single vCenter controlling multiple offices, servers, the conversation you get into at that point is, how many Veeam servers do you deploy? Where do you deploy them? How do you connect into those ESXi hosts and the vCenter? And the conversation usually comes down to something along the lines of, for vCenter, VMware vCenter is your brain. And the Veeam backup and replication server is kind of the same thing. If you've architected everything correctly, not everything goes through your brain. Your brain just tells the other organs what to do. What are those other organs on the VMware side? That's your storage and your ESXi hosts. On the Veeam side, that might be your proxies and repositories. And if you've architected your backup environment correctly, you don't send all of your traffic when you do a backup or restore through the vCenter and the Veeam backup and replication server. That goes down a level where it flows directly from your proxies and repositories to your storage and compute, which means if you have, say, a local proxy and a local repository at your local office, you can keep your local copy of your backup that doesn't traverse across your internet link across offices. Because that really is what the primary concern tends to be. Hey, are we going to put in a local repository or a local Veeam server, and we don't want the traffic to go back to the main office where the vCenter is, and then back to that remote office? We want it to just back up locally from local host to local Veeam server. Yeah, fair enough. And I think, Maurice, you mentioned the VM move between hosts earlier. So have you ever had that, the VM move between host and then suddenly it just showed up as a new VM in backups? No, because I don't add the SXI host directly unless I need to restore vCenter. I tend to connect vCenter to ensure that this won't happen, to prevent things like that happening. Yeah, I have not personally done it, but I have seen people do it before. And I don't know that it necessarily has changed the ID. It's been a number of years since I looked, but what it did cause was backup failures. Because obviously, if the backup job is pointed to a single host, if the VM moves over to another host, you can't see the VM anymore to back it up. And then sometimes it might move back and autocorrect. Sometimes it might not. How do you handle that? It usually ends up being a conversation with the customer that we need to move over to using their vCenter. Usually, thankfully, we catch it pretty early as just part of our onboarding checklist. But it can be a little frustrating because it does usually, the change over to best practices and properly backing things up can cause new backup chains just because of changing the reference from the vCenter to that host. Because again, it's the brain. So you change the brain, now you're using the vCenter instead of the host. Some things like mentioned in the article, like the VM ID might change or other certain mappings that Veeam requires in order to know what machine goes to what restore point. Because Veeam can't just magically go, oh, this is the machine. It goes to this restore point. It uses things like those MoRef ID. No, okay, I think great point in here. And I don't know if this is the right conclusion, but you can help me to build on this conclusion of this great article shared by Andreas. But like the standalone host can work in a small and static setup, but like for most environments, what you are saying it like vCenter, it makes your life much easier, right? Right, and really it comes down to, you should be talking to your production environment at the highest layer that you have available. So if you have a cluster, you should be talking to the cluster, not just the standalone host. Or if you have a vCenter, you should be talking to a vCenter to talk to the cluster. Okay, pretty cool. Thank you, Andreas for that. And I have to also appreciate all the images created with AI. They are pretty cool. I'm sure everyone in this call appreciates that. So kudos for that. Awesome, anything else that you want to add for that article or should we just move forward with the third one? No, I think we can move to the third one. So you were mentioning forensics analysis. So I think this is where we gonna talk a little bit about that topic. So this is an article from Eric. Great to have you back in the recap. He became a regular as well, great job. This is a really interesting topic from my point of view. And what he's doing, he's diving into it. The title says, Veeam Failure Forensics Reconstructed What Went Wrong From Session Logs. Pretty original as well. I haven't really seen much discussion around it in the community hub. So I think that's fantastic. And he's not talking only about just, fixing a job that failed. It's about going into your history session logs and figuring out when problems started, what was affected, proving it with evidence. And then he mentions a few points as we can go over the article, like checking session and task data, analyzing bottlenecks, finding silently failing or unprotected VMs and correlating with alarms to get the full picture. What do you think about this topic? And how would you, or how do you tackle these investigations in your environments? Because I'm sure you guys do it at your companies. I think one of the important things in my book is to ensure that the log files are not on your VBR server. At least you have a copy of those log files in a Syslog server or a SIEM server or whatever you are using. Because that would ensure that in case your VBR server is compromised, you still have that data because getting it out of that SIEM server should be harder than deleting a bunch of files from your file system. So that's in my book is one of the more important things in general to ensure you can do your forensics at one point. Other than that, yeah, I think it's a really good write-up on how you can do it and what you can do with the VBR server itself. Yeah, I really like it because this really takes me back to maybe the end of my first year, beginning of my second year as a service provider. I was going through, I had recently completed the VMCE. I was being sent through the VMCA. I think it was back in version one of the VMCA, but before we, years were put at the end of the VMCENA as the certifications and it was mapped to specific Veeam versions. And one of the critical parts of going through the VMCA class that was probably most useful to me then and still is to this day with Veeam is learning where all the logs live and how to read the Veeam logs. Because even these days, being able to pull the Veeam logs, I can go feed those into an LLM like chat GPT, but it's probably 50-50 whether it actually tells me what the error is and what's going on. Because there can be a lot of cases where maybe there's a small temporary network where Veeam retries and it succeeds and the error is something else on say a login to vCenter. And I liked that Veeam one was brought up in this too because going in and looking at your logs and fixing a backup error, you might just be addressing the symptom and not the root cause, right? I remember first building my home lab, I went out and I got the VMware user group license, the VMUG Advantage license, I spun up a VMware server, I had a storage server and I was having all sorts of random issues with backups or even accessing my machines. And I correlated it to when backups were running, but I couldn't quite figure out what was going on. Bringing in Veeam one and seeing some of the Veeam one alarms like, hey, your data stored latency is too high is what finally helped me track down like, oh shoot, I'm doing too many tasks on the production repository, I'm stunning things. You know, I need to limit my latency or my performance throughput so that I'm not stressing my VMware data store so hard versus before that I didn't know, was it the compute host? Was it the data store? Was it something on Veeam's side that was getting stuck? All of it together is what finally led to me finding that root cause and addressing it. Okay. Can I ask you if you ever discovered like, you know, a backup problem, just like weeks after it started or a month after it started and how did you track it down? What did you do? Ideally, you have monitoring in place to verify that your jobs are running either via Veeam one, via a script, via logging into your backup server every morning, checking if your jobs were running okay and if there were failing jobs. I wouldn't suggest the last one, but not ideally. That's ideally, right? Well, the last one is not really ideally because that means that you need to work seven days a week because your jobs run seven days a week if done correctly. So ideally you have monitoring in place either via Veeam one, via a separate monitoring, via emails that are sent out to your mailbox for failed jobs. But you need to make sure that you're actually checking those things because I will say one of the number one things I have seen as a service provider that have caused headaches with recovery, you know, kind of even going back to the first article is, you know, we set up a reporting system where a customer gets a daily email or an email where the job completes and they set up a rule where it folders that. And what happens for six or eight weeks, their backups are broken. We're telling them they're broken, but they're not seeing that email. And then all of a sudden they happen to log in to their Veeam server, see it's broken, and they call our support, all sorts of angry and upset and not understanding what's going on and why we didn't reach out to them. And it's like, we have been reaching out to you for eight weeks, you haven't been getting back to us. Yeah. Yeah. Yeah. Like I said, good monitoring, ensuring that your jobs are running, but also verifying that you can restore them. Obviously, those steps all together, yeah, I think that those are important. Yeah, absolutely. I totally agree with that. And let me ask you one final question as we are at this article, because it's too good. So is there any point in here that seems to you very, very important when we talk forensics in this article? I like the silent protection gaps are worse than failed jobs. You know, that's a very big thing that I think everyone struggles with. How do you make sure that everything that needs to be protected is protected, especially when you don't know what you don't know? Fair enough. Maurice, anything else that stands out for you? No, I think knowing what your backup and knowing that you can restore those backups and having a decent plan is really important, yeah. Awesome. Great article. Thank you again, Erik. Just keep on posting. We love your articles. And now we're finished with the articles and we are moving to the fun conversation, Hackathon. So why don't you just, you posted today this article at Automation Desk, Maurice. Why don't you take us through some important information when it comes to this year, Hackathon? Yeah, we're doing another round of the Veeam Community Hackathon this year. It's in October. I hope, we really hope that this is a better time window for the most people. December last year was not the best period maybe. And we shifted a few weeks from end of September. That way, we're at the beginning of the quarter instead of the end of the quarter for those of you like me who have quarterly projects that are maybe coming up to the finish line in the last few weeks of the quarter. Exactly. So yeah, we are doing it again for the fourth time this year. So that makes us, yeah, made us create a web application this time around for you to sign up instead of a form. If you have signed up in any of the previous years, the majority of your data is already there. It's a pretty quick and simple signup form. You just fill in your full name, your email, create your password. Your time zone should be detected for your browser. And you agree to the code of conduct and you'll get an email in your mailbox, sometimes in your spam folder, unfortunately. But yeah, you verify your email, you log in and you finish your profile. The majority of the details are already there. Your shipping address is something we could not recover, unfortunately. But yeah, we created this application for everyone to enjoy and sign up. Once you finish your signup, you can even invite up to 10 other people to join as well. I think this is great. I love this new format, Maurice, Jonah. I think it's just getting better and better every year. And I do hope that this year, we're gonna have more success with the registrants than last year. Unfortunately, some people got sick and some people couldn't make it. Maybe it was, as you said, like not the best time of the year. December, everyone is kind of, with their mind switched off and they think about Christmas already and New Year. So this year, I'm glad that you moved it to September really. And yeah, hopefully we're gonna have a great one. But why don't you tell us as well a little bit about the event series that you prepare for Automation Desk? Yeah, sure. I do want to mention one little thing about this application. Once you have finished your signup, you will get some calendar invites as well that will block your calendar for the hackathon directly. So it's in your agenda at that point. Yeah, that should make the signup process a lot less cumbersome, especially for those of you who have participated for multiple years. There's no more having to fill out the form all over again from scratch every year. You just have to come in, maybe add one or two more things as we add them in the future. Like you can see, for example, here on the screen under product experience, Beam Data Cloud for Microsoft 365 is required because that's a new category under product experience because we added the Beam Data Cloud suite for the experience ratings. But in the future, most or all of these should say ready when you log in immediately. You just need to quickly verify your information is correct. And then you can sign up with one or two clicks as opposed to redoing that entire form like in previous years. Exactly. Why should people participate to this event? It's fun, for starters. We know that. No, but aside from it being fun, it could help you kickstart your way into a different role or a different experience that you didn't expect you would be doing on a daily basis. But once you, well, tip your toes in, basically. So if you are not a coder, basically, you kind of have the opportunity to work maybe with some coders and kind of learn maybe a little bit of coding. Or if you are, you want to kind of improve your public speaking, you never had the opportunity to do a presentation, you can just record at the end the presentation, you can be the presenter or you can be the team leader for the team, so you can coordinate. So you can get in different roles, right? And you can just have the opportunity to make it fun, meet new people, be part of a community and also kind of learn new skills. So I think it's great. The only requirement that we have is that you're 18 years or older at the start of the event. So if you're 17 today, but you will be 18 at the point of the event, then you can join as well. And you don't have to have Veeam experience, right? No, no. Learning on the job, right? Yeah, that's what it's all about. I mean, we've seen multiple times in previous years where people who have no Veeam experience, maybe they're even still in their college or university have signed up for the event just as an opportunity to enhance their existing programming skills, start learning the IT market a little bit. And they ended up meeting people on their teams who then actually ended up hiring them later in the Veeam space as developers or other roles. Very cool. But it could also be that you don't have any experience with Veeam One, but you are experienced with VBR, Veeam Backup Replication, and your team decides from working on Veeam One or you as a team decide to work on Veeam One, but it gives you an opportunity to learn Veeam One as well. Yeah, that's awesome. I mean, yeah, there's really no reason for not joining. I think everyone that has an interest to understand a little bit more what a hackathon is should just join us and have some fun with us. And to get you started, we are working on a hackathon one-on-one series, including a automation to spotlight to get you on the road and understand a bit what you can do and how it works. And two weeks ago, I think we had this first event or this first webinar to set up your dev environment to show you a little bit on what you can do, what you need to do, how you can create your dev environment and get started directly. So where can we find this one? Is it like on your YouTube, on the hackathon YouTube? Do we have it, Sofia, as well on the- It's on the Veeam YouTube. It's on Veeam's YouTube. Yeah. Oh, okay. That's cool. We're gonna share a link to it, so you have the chance to watch it. There's a Veeam Automation Desk YouTube as well that as we build these up, we're going to be working to take them plus some historical recordings we have and build a hackathon toolkit playlist, at least for the one-on-ones. So the thinking is the hackathon one-on-ones are your toolkit of your essential things you need to know, especially if you're looking to do more development on the hackathon side of things leading up to the event. And then the community spotlight is a way in which we're trying to help showcase just interesting and neat automation-related projects that we see out in the community. So you can see like in May, we're talking to get potentially Jorge on a webinar with us to talk about his Grafana dashboards. And we have a few other ideas in the pipeline. And the hope is to just maybe showcase some neat projects that might benefit people in their day-to-day jobs and also provide inspiration of the sorts of things that you might try and have teams work on during the hackathon itself. Yeah, and this month you will see a couple of blog posts on the automation desk about open source licenses. So yeah, keep the automation desk at a close loop to see when they are coming. Yeah, if you are not subscribed, you can always just go and subscribe for the automation desk so you can get the notifications on your email. Exactly. Get a team together, people. You have a lot of time though. Sign-ups do close in September, which will probably be here tomorrow before we know it with how time is flying. But you still have time, but it'll be September before we know it. Yeah, we still have a lot of things to do, Sofia. So don't get my heart racing here. Okay. Well, the point is like you're gonna hear a lot about the hackathon from now on. Also, Sofia, we are having a special edition Vim 100 show, right? Yeah, that's the plan to have the winners. We've been doing that for the past couple of years, having the team projects. This year we'll have the winner sometime in July or August. So stay tuned for that. Awesome. Awesome. Good stuff. Anything else that you would like to add to the hackathon discussion, guys? Otherwise we're gonna move to the special department news. No, I don't think I have anything to add other than sign up now. They're open. That's pretty much it. Sign up now. Sign up. Okay, let's go for the special department news, Sofia. Yeah, so we have a few events to highlight as well as blog of the month competition for World Backup Day. But I wanna kick off by talking about a new Vim user group in Milwaukee, Wisconsin. So we have Brendan over there kicking off strong. We literally spoke to him for the first time probably not even two weeks ago, and he's already hosting his first event with Everpure. It's a collaboration with him, I know. He moved fast. So I'm looking forward to hearing about how this event goes. But if you're in the Milwaukee area on April 14th, so just a few days from today, they'll be having an afternoon meetup with Everpure sponsoring, like I mentioned, and some Vim SEs local to the area will be there as well as a systems engineer from Everpure. And then they'll end it off with networking in an happy hour, just in that last hour of the day. So if you're in the area, definitely make sure to attend. I'm looking forward to hearing about it from Brendan afterwards. Absolutely, yeah. This is pretty cool. I saw all his posts about this as well on LinkedIn and social media. So good job to you, Brendan. Absolutely. Next up, we have a user group over in Germany. Again, just a few days from today on April 15th, next Wednesday, so a week from today. And if we go to the registration page, in true most of the European fashion, it'll be an all day affair for the user group compared to the Wisconsin one just a little bit ago. So it'll be a longer day, but the Vim user group will be in Cologne, that region. I think this is the first one in that area, at least the first one that I've been aware of that they've hosted in this region, but they'll have Scality there, as well as some Vimers who will be presenting. Here are some of the speakers just for reference. So Hannes will be there. We have Vim Vanguard Mateus, Marvin, who is a new Vim MVP. And then they'll have Rudiger and Niels, excuse me if I'm pronouncing that wrong, from Scality, as well as Falco, the Vim user group leader presenting. Let me turn on translation so I can actually read the content of the day. But yeah, so they'll have next level backup with Scality and Vim, VB365 and VDC What's New, topic on deep dive object memory from Falco, VBR V13.1 and VHR What's Up session from Hannes, and a surprise session. So I don't know who's speaking. I think they titled him, they wanted to add him to the speaker list as like mystery ninja or something. So I have really no idea what that will be. I'm curious what we'll find out after who that special guest of the event is that they're keeping a surprise. I guess we'll have to see. And a presentation from Mateus on SOBR. And then an AMA, of course, at the end of the day, in true German fashion always, they always have like someone from the product team. They're very lucky with getting that support there. And it'll be from Hannes and Marvin. So few days to sign up. Only, I think they only have about 10 spots left. It's filling up pretty quickly. So if you're in the area and you happen to see this video before the event next week, make sure to sign up because there's not that much space left. Like last time when we checked the registration were like 65 people that registered and they were targeting like 70, right? Yeah, I think they have 65 spots and they're letting 70 register. Just, you know, we all know how events operate with people not showing up, but they're nearly there. I think this morning I checked, it's about 58 people registered or something along those lines. So, you know, it's always a popular user group with the German team there. So hopefully. Great job, Michael, Falco, Tobias. Absolutely. Okay. Last special department news and I'll also allow Maddie to speak to this, but we have the World Backup Day Contest for a special blog contest that we're doing. Now is the time to vote for the winner. So Maddie, tell us about the responses that you got to this. Oh, I think we actually got like about 12 articles, really good ones, strong topics in there. Thank you all for participating and really wish you good luck. You all received that awesome badge that I really like. And yeah, just all of you help us to get the winners because we have a first place. It's gonna be the most votes in here. We have a second place, most likes on LinkedIn posts and the third place, most likes on community hub posts. So. When does voting close, Maddie? It's going to be next week. I believe it's the 14th, if I'm not wrong. I think I mentioned it somewhere here. Yeah, it should be there. Go down, scroll down a little bit. Yeah, there a little bit. Yeah, April 14th. Yeah, so you still have plenty of time to vote. Maurice, Jonah, did you pick your favorite already? I have not yet. Me too. Later today. Let's vote. I think they all deserve, in my eyes, they all deserve, obviously, to win, but the community ultimately is going to, you know, to vote and they're gonna decide who are the winners. But thank you all really. I mean, it's always appreciated when you contribute and you kind of help the community with, beside the contest, you know, like in the prizes and all that, sharing is caring and you are helping the whole community, with your stories, with your experience, with your learning. So I think that makes everyone a winner, in my eyes. Absolutely. Yeah. Good luck, everyone. Looking forward to seeing all the winners. Okay, and we'll cap it off in true community recap style with the Who's New, if I'm allowed to make it full screen. Okay. But this past week, we've had 127 people join the community. So welcome, everyone. I did pick the four most interesting usernames I found in that 127. So special welcome to Thor OMG. I think they're a Thor fan, I would assume. Chrome Heart JP, Andrew One Million, Matty 3D, and Alfred's pick, Sparky Marco Polo. Yeah, that's interesting. So welcome, everyone. This is great to have always. It's a plus hundred. So that's amazing. I hope, you know, everyone enjoyed this episode. I know it's gonna be a little bit longer than normal, but I think we all had some fun discussions in here. And I hope it's gonna be useful for everyone. Thank you for putting together the hackathon this year, Jonah, Maurice. Let's talk more about it in the next few months. I'm sure we will. And always fantastic to have you as a guest. Thank you for taking the time. And anything else that you would like to add? Sign up for the hackathon. I know, that's what you were supposed to do. That was your call. Okay, fantastic. Thank you all for watching. And we'll see you next week with a new episode of The Recap.

TL;DR

  • Immutable backups are essential but insufficient alone—comprehensive protection requires multi-layered security, redundant copies, MFA, and regular recovery testing to ensure data is both protected and recoverable
  • Understanding Veeam's logging infrastructure is critical for effective troubleshooting, as backup errors often represent symptoms rather than root causes, and tools like Veeam ONE can reveal underlying infrastructure issues
  • The fourth annual Veeam Community Hackathon moves to October with a new web application for registration, avoiding year-end conflicts and making participation easier for the global community
  • Proper monitoring systems are worthless if administrators don't review alerts—automated email notifications that get filtered to folders have led to customers discovering backup failures weeks or months after they began
  • Silent protection gaps where workloads aren't being backed up at all are more dangerous than failed backup jobs, making comprehensive inventory and validation essential

Why Immutability Isn't Enough for Data Protection

The episode opens with a deep dive into Ken's community article exploring why immutable backups, while critical, represent only the first layer of a comprehensive data protection strategy. The discussion emphasizes that immutability protects against deletion but doesn't guarantee recoverability. Jonah shares real-world examples of compromised S3 environments where immutable data remained technically intact but became inaccessible due to file versioning attacks. The conversation stresses the importance of multi-layered security including MFA, redundant copies, and regular recovery testing. Maurice reinforces that keeping software updated and maintaining current patches is essential, as every piece of software will eventually have vulnerabilities. The panel agrees that testing restore capabilities is more important than simply verifying backup completion, as silent protection gaps are worse than failed jobs.

Forensics and Troubleshooting with Veeam Logs

The discussion shifts to Erik's article on incident forensics and the critical importance of understanding Veeam's logging infrastructure. Jonah reflects on his VMCA training experience, noting that learning where logs live and how to read them remains one of the most valuable skills for troubleshooting. The panel discusses how addressing backup errors often treats symptoms rather than root causes, with Jonah sharing a personal example of using Veeam ONE to identify datastore latency issues that were causing seemingly unrelated backup problems. Maurice emphasizes the importance of exporting logs to external SIEM systems to ensure forensic data survives if the VBR server is compromised. The conversation highlights that proper monitoring—whether through Veeam ONE, scripts, or email alerts—is essential, but only if administrators actually review the notifications rather than filtering them to folders.

Veeam Community Hackathon 4th Edition Announcement

Maurice and Jonah announce the fourth annual Veeam Community Hackathon, scheduled for October to avoid the year-end conflicts that affected the December event. This year introduces a dedicated web application for registration, replacing the previous form-based system, making it easier for participants to sign up and track their progress. The hackathon continues to focus on automation and innovation within the Veeam ecosystem, bringing together community members to collaborate on creative solutions. The panel encourages community members to register early and participate in what has become a signature community event. Additional user group events are highlighted, including upcoming meetups in Wisconsin and Cologne, Germany, with the German event nearly at capacity with 58 of 70 spots filled.

Chapters

0:00 - Episode Introduction & Guest Introductions
2:26 - Beyond Immutability: Multi-Layered Protection
9:23 - Recovery Testing & Backup Validation
21:15 - Incident Forensics & Log Analysis
26:53 - Veeam Community Hackathon 4th Edition
38:10 - User Group Events: Wisconsin & Germany
41:44 - World Backup Day Contest Voting
44:12 - Who's New & Episode Wrap-Up

Key Quotes

4:55 "I have seen corrupted backups before where they're still there in multiple methods, including it's not very common, but I've seen multiple instances out there in the wild where a customer actually has immutable backups with an object storage vendor and the data is all still there, but the bad actor has essentially managed to compromise the Windows Server that Veeam was running on extract the S3 keys."
6:02 "Immutability is only at the storage layer. There's still other layers that social engineering can occur or hardware failures can occur. Resiliency is important and security is important."
8:05 "Keep in mind that every piece of software will have a vulnerability at one point, so keep it up-to-date. Don't Slack with that."
22:05 "Even these days, being able to pull the Veeam logs, I can go feed those into an LLM like chat GPT, but it's probably 50-50 whether it actually tells me what the error is and what's going on."
24:43 "One of the number one things I have seen as a service provider that have caused headaches with recovery, you know, kind of even going back to the first article is, you know, we set up a reporting system where a customer gets a daily email or an email where the job completes and they set up a rule where it folders that. And what happens for six or eight weeks, their backups are broken."
26:04 "Silent protection gaps are worse than failed jobs. You know, that's a very big thing that I think everyone struggles with. How do you make sure that everything that needs to be protected is protected, especially when you don't know what you don't know? ..."

FAQ

Why aren't immutable backups enough to protect my data?

Immutability only protects against deletion at the storage layer. It doesn't prevent corruption, ensure recoverability, or protect against compromised credentials being used to manipulate file versions or metadata. A comprehensive strategy requires multiple layers including MFA, redundant copies in different locations, regular recovery testing, and monitoring to detect silent failures.

How can I make sure my backup monitoring doesn't get neglected?

Implement automated monitoring through Veeam ONE, scripts, or email alerts, but critically ensure these notifications aren't being filtered to folders and ignored. Consider integrating with ticketing systems or SIEM platforms that require acknowledgment. Regular recovery testing should be scheduled and tracked, not just backup completion verification.

What's the most important thing to learn for troubleshooting Veeam issues?

Understanding where Veeam logs are stored and how to read them is essential. The VMCA certification covers this comprehensively. Knowing log locations helps identify root causes versus symptoms—for example, a backup error might actually be caused by underlying storage latency issues that only become apparent when reviewing multiple log sources including Veeam ONE alerts.


Categories:
  • » Data Protection » Backup & Recovery
  • » Data Protection
Channels:
News:
Events:
Tags:
  • Data Protection
  • Best Practices
  • Technical Deep Dive
  • Backup & Recovery
  • Security Operations
  • Community Event
  • Immutable Backups
  • Data Protection Strategy
  • Backup Testing
  • Incident Forensics
  • Veeam Logging
  • S3 Object Storage Security
  • Multi-Factor Authentication
Show more Show less

Browse videos

  • Related
  • Featured
  • By date
  • Most viewed
  • Top rated
  •  

              Video's comments: Veeam: Beyond Immutability: Backup Testing & Forensics Best Practices

              Industry Events (Sponsor Hosted)

              • Sep
                17

                Bridging the SaaS Protection Gap: Preventing Data Loss and AI Missteps

                09/17/202610:00 AM ET
                • Sep
                  17

                  Unveiling the AI-Driven Underworld of Automation's Rapid Rise

                  09/17/202601:00 PM ET
                  • Sep
                    23

                    Invisible Data: Understanding What You Can’t Safeguard

                    09/23/202601:00 PM ET
                    More events

                    Upcoming Webinar Calendar

                    • 09/17/2026
                      10:00 AM
                      09/17/2026
                      Bridging the SaaS Protection Gap: Preventing Data Loss and AI Missteps
                      https://www.truthinit.com/index.php/channel/2119/bridging-the-saas-protection-gap-preventing-data-loss-and-ai-missteps/
                    • 09/17/2026
                      01:00 PM
                      09/17/2026
                      Unveiling the AI-Driven Underworld of Automation's Rapid Rise
                      https://www.truthinit.com/index.php/channel/2108/unveiling-the-ai-driven-underworld-of-automations-rapid-rise/
                    • 09/23/2026
                      01:00 PM
                      09/23/2026
                      Invisible Data: Understanding What You Can’t Safeguard
                      https://www.truthinit.com/index.php/channel/2087/invisible-data-understanding-what-you-cant-safeguard/
                    • 09/29/2026
                      12:00 PM
                      09/29/2026
                      Embracing AI Adoption While Ensuring Robust Security Measures
                      https://www.truthinit.com/index.php/channel/2092/embracing-ai-adoption-while-ensuring-robust-security-measures/
                    • 09/30/2026
                      04:00 AM
                      09/30/2026
                      AI Command Center: Optimizing Visibility and Control in Your Operations
                      https://www.truthinit.com/index.php/channel/2024/ai-command-center-optimizing-visibility-and-control-in-your-operations/
                    • 11/19/2026
                      01:00 PM
                      11/19/2026
                      360View: Govern, Secure & Recover Your Microsoft 365 Environment
                      https://www.truthinit.com/index.php/channel/2076/360view-govern-secure-recover-your-microsoft-365-environment/
                    Truth in IT
                    • Sponsor
                    • About Us
                    • Terms of Service
                    • Privacy Policy
                    • Contact Us
                    • Preference Management
                    Desktop version
                    Standard version