This seems to be a popular search term so I think it’s worth covering off. This is covered on my old top post about Fractional Reservation, but I’ll cover the alternatives here also.
NetApp snapshots used to be pretty unique in the industry, but the industry term for this technology is generally now Append-on-Write / Redirect-on-Write (new writes are appended to the “end”, or redirected to free blocks, depending how you look at it) and quite a few vendors do it this way. Put very simply, all new data is written to new (zeroed) blocks on disk. This does mean that snapshot space has to be logically in the same location as the production data, but that really shouldn’t be a problem with wide-striping / aggregates / storage pools (pick preferred vendor term). When a snapshot is taken, the inode table is frozen and copied. The inode table points to the data blocks, and these data blocks now become fixed. As the active filesystem “changes” blocks, these actually get written to new locations on disk, and so there is no overhead to the write (the new blocks are already zeroed). In other technologies (not NetApp) this also forms the basis of automated tiering, once data is “locked” by a snapshot, it’ll never be over-written so it can safely be tiered out of SSD or even SAS as read performance is rarely an issue. NetApp use FlashPools to augment this, and a snapshot is a trigger for data to be tiered out of FlashPools as it’ll never be “overwritten”.
First new post in what, 2-3 years? WAFL.co.uk still performs admirably and now that I’ve moved roles I figure it’s time to re-visit my old flames. My home lab needs a re-build and upgrade and that should get documented!
So I’ve moved into the big scary world of contract based work, I started my first role a couple of weeks ago, and so far it’s going great. I want to keep an update on my exploits, the challenges real customers are facing and share some of my generic musings. The site will be less NetApp centric, but I still have my roots in storage!
My first role involved a lot of interesting challenges, but there’s some great technology available here too. A strong DevOps team (that need help integrating the Ops bit, doesn’t everyone?), lots of Big Data challenges, and an immediate project to look at creating a much more responsive infrastructure, including where cloud services fit in. I started life as a web developer, and it’s great being back at a dotcom company and seeing how the challenges have evolved.
NetApp last week released OnCommand Performance Manager 1.0 Release Candidate 1 (RC1) to all NetApp customers and partners. This new software provides performance management, troubleshooting, and event notification for systems running clustered Data ONTAP 8.2 and 8.2.1.
Performance Manager 1.0 RC1 is deployed, not installed, as a virtual appliance within VMware ESX or ESXi. A virtual appliance is a prebuilt software bundle, containing an operating system and software applications that are integrated, managed, and updated as a package. This software distribution method simplifies what would be an otherwise complex installation process.
Upon deployment, the Linux 2.6.32-based virtual appliance creates a virtual machine containing the user software, third-party applications, and all configuration information pre-installed on the virtual machine. Much of the virtual appliance middleware is built primarily with Java and includes several open-source components – most notably from (but not limited to) the Apache Software Foundation, the Debian Project, and the Free Software Foundation.
Sizing Performance Manager is based upon a number of factors: the number of clustered Data ONTAP clusters, maximum number of nodes in each cluster, and maximum number of volumes on any node in a cluster.
In order to meet the official supportability status from NetApp, Performance Manager 1.0 RC1 requires 12GB of (reserved) memory, 4 virtual CPUs, and a total of 9572 MHz of (reserved) CPU. This qualified configuration meets minimum levels of acceptable performance and configuring these settings smaller than specified is not supported. Interestingly, increasing any of these resources is permitted – but not recommended – as doing so provides little additional value.
In fact, according to December 2013 AutoSupport data from NetApp, most customers should expect to deploy a single Performance Manager virtual appliance; as one instance will be suitable for 95% of all currently deployed clustered Data ONTAP systems.
NetApp today launched the FAS8000 Series, its latest enterprise platform for shared infrastructure, with three new models: FAS8020, FAS8040, and FAS8060, which replace the FAS/V3220, FAS/V3250, and FAS/V6220, respectively. This new line will initially ship with Data ONTAP 8.2.1 RC2, supporting either 7-Mode or clustered Data ONTAP.
All systems are available in either standalone and HA configurations within a single chassis. All standalone FAS8000 controller configurations can have a second controller (of the same model) added to the chassis to become HA.
The new FAS8000 has been qualified with the DS2246, DS4246, DS4486, DS4243, DS14mk4, and the DS14mk2-AT disk shelves with IOM6, IOM3, ESH4, and AT-FCX shelf modules. Virtualized storage from multiple vendors can also be added to the FAS8000 — without a dedicated V-Series “gateway” system — with the new “FlexArray” software feature.
NetApp will not offer a separate FlexCache model for the FAS8000 Series.
Let’s explore the technical details of each one of these new storage systems.
The 3U form factor FAS8020 (codenamed: “Buell”) is targeted towards mid-size enterprise customers with mixed workloads. Each Processor Control Module (PCM) includes a single-socket, 2.0 GHz Intel E5-2620 “Sandy Bridge-EP” processor with 6 cores (12 per HA pair), an Intel Patsburg-J SouthBridge, and 24GB of DDR3 physical memory (48GB per HA pair).
NetApp supports single and dual controller configurations in one chassis, but unlike previous systems, I/O Expansion Module (IOXM) configurations are not supported. The increased mix of high-performance on-board ports and the flexibility offered by the new Unified Target Adapter 2 (UTA2) ports reduces the need for higher slot counts on the FAS8000 series.
But even if you didn’t get excited about the big game, you might still be interested to learn about the role of data storage for the most-watched television program in American history.
During the week leading up to the Super Bowl, I had the privilege to help ring the opening bell at the NASDAQ MarketSite in New York City — and what an experience! I also had the opportunity to chat with the NFL’s Director of Information Technology, Aaron Amendolia, to explore how they leverage NetApp storage systems for data management.
It starts with 40 NetApp FAS2200 Series storage systems that store, protect, and serve data to all 32 NFL teams, thousands of personnel, and millions of fans. For example:
Want player stats during the game? All game play raw data is instantly available and served by NetApp storage systems.
Like those action shots? Television and newspaper photographers take hundreds of thousands of photos and videographers capture high-definition video of regular-season games, the playoffs, and the Super Bowl – all stored on NetApp storage systems.
See someone wearing a badge? NetApp provides the infrastructure that supports security credentialing for everyone from hot dog vendors to the NFL commissioner.
I also learned that the NFL leverages the entire protocol stack (both SAN and NAS), with over 90% of their infrastructure running virtual machines on NetApp storage systems.
Yet, every Super Bowl is unique.
The NFL’s end-users are often located in hotels with high-latency connections; hardware is subjected to harsh environments usually not found within most datacenters (soda can spills, dirt, grit, etc.). The good news is that SnapMirror, the replication software built into Data ONTAP, allows the NFL to failover in the event of a problem.
NetApp today announced the availability of Flash Accel 1.3.0, its server-side software that turns supported server flash into a cache for the backend Data ONTAP storage systems.
As with previous releases, Flash Accel 1.3.0 detects and corrects for coherency at the block level — rather than flushing the entire cache. Flushing the entire cache may be good as there are no data coherency issues, but terrible for performance. Flash Accel cache invalidation corrects cache, while keeping the cache persistent.
In addition to intelligent data coherency, Flash Accel also provides persistence across VM / server reboots.
The benefit of both intelligent data coherency and persistence is to ensure that both the cache optimizes performance at its peak (i.e. when the cache is warm) and that peak performance can last as long as possible (by keeping the cache warm for as long as possible).
Side note: Flash Accel code manages server cache, accelerating access to data that is stored and managed by Data ONTAP on the storage system. Flash Accel is NOT Data ONTAP code.
Flash Accel 1.3.0 adds the following features and functionalities:
- Support for Windows Server bare metal caching:
- Windows 2008 R2, Windows 2012, and Windows 2012 R2
- FC and iSCSI support for bare metal
- Clustered apps supported (cold cache on failover)
- Adds support for Windows 2012 and 2012 R2 VMs and vSphere 5.5 support
- Note: for use of Flash Accel with Flash Accel Management Console (FAMC), vSphere 5.5 support will be added within weeks of general availability of Flash Accel 1.3
- Up to 4TB of cache per server
- Support for sTEC PCI-e Accelerator
- Note: For VMware environment, Flash Accel 1.3.0 is initially only available for use with FAMC. 1.3.0 support for use with NetApp Virtual Storage Console (VSC) will be available when VSC 5.0 releases
NetApp Snap Creator Framework is data protection software that integrates NetApp features with third-party applications, databases, hypervisors, and operating systems.
Snap Creator was originally developed in October 2007 by NetApp Professional Services and Rapid Response Engineering to reduce (or even eliminate) scripting. Nowadays, Snap Creator is a fully supported software distribution available from the NetApp Support Site.
The Snap Creator Team provides two versions of Snap Creator: a community version and an official NetApp release. The community version includes the latest plug-ins, enhancements, and features but is not supported by NetApp Support. The NetApp Version is fully tested and supported, but does not include the latest plug-ins, features, and enhancements.
Let’s explore the architecture of the recently released Snap Creator 4.1 Community Release in November 2013.
Snap Creator Server Architecture
The Snap Creator Server is normally installed on a centralized server. It includes a Workflow Engine, which is a multi-threaded, XML-driven component that executes all Snap Creator commands.
Both the Snap Creator GUI and CLI, as well as third-party solutions (such as PowerShell Cmdlets), leverage the Snap Creator API. For example, NetApp Workflow Automation can leverage PowerShell Cmdlets to communicate to Snap Creator.
To store its configurations, Snap Creator includes configuration files and profiles in its Repository; this includes global configs and profile-level global configs. If you’re familiar with previous versions of Snap Creator Server, one of the new components is the Extended Repository. This extension provides a database location for every job, imported information about jobs, and even plug-in metadata.
For persistence, the Snap Creator Database stores details on Snap Creator schedules and jobs, as well as RBAC users and roles.
Since it’s the season of countdowns and top 10 lists, there’s no better way to look back on 2013 than a list of this year’s best products and solutions from NetApp. Without further ado, I give you my NetApp “Top 10″ of 2013:
1. All-Flash Arrays (EF-Series, FlashRay)
Without a doubt, all-flash arrays were the biggest story of the year; not just one, but two! The launch of the EF540 and announcement of the FlashRay kicked things off in February 2013, followed closely with the EF550 launch nine months later in November. The EF-Series delivers consistent, predictable, sub-millisecond response times to accelerate the latency-sensitive applications responsible for driving revenue, productivity, and/or customer satisfaction on a day-in, day-out basis.
2. Clustered Data ONTAP
June 2013 ushered in the launch of the NetApp flagship operating system, clustered Data ONTAP 8.2. Its focus is continuous data access during controller upgrades; granular quality of service (QoS) for SAN and NAS; Storage Virtual Machines (SVMs) that deliver storage services; and support for the latest storage protocols including SMB 3.0, pNFS v4.1, iSCSI, FCoE, and Fibre Channel.
2013 also marked another ‘first’ for Cisco and NetApp: FlexPod became the #1 Integrated Infrastructure by IDC. The Cisco and NetApp partnership that brings the FlexPod systems to market generated $203.1 million worth of “integrated infrastructure” sales during 2Q13, representing a 26.2% share and the top ranking within this market segment.
4. Cloud Strategy
In 2013, IT slowly began the transition from being builders and operators of data centers to becoming brokers of application and information services. To address the gap of blending public and private clouds, NetApp’s strategy is to use the world’s number-one branded storage operating system, Data ONTAP, as a universal data platform across cloud environments. It is now part of over 175 service providers throughout the world!
NetApp last week released Data ONTAP 8.2.1 Release Candidate (RC) 1, an update to its flagship operating system for FAS/V-Series Storage Systems. The new features of version 8.2.1 focus on three key areas:
- Shelf Removal
- Storage Refresh Automation
- MS SQL Server over SMB 3.0 with continuously-available shares
- Offbox Antivirus
- Automated Workflow Analyzer (AWA)
- Data ONTAP Edge for clustered Data ONTAP
- System Setup
- LDAP over SSL
- Enterprise SMB features
- Qtree export policies for NFSv3
- Increased SnapMirror and SnapVault fan-in
- In-place 32-bit to 64-bit aggregate upgrade
In addition to these features, the OnCommand Management Suite has been updated to support this release with the following new release candidates:
- System Manager 3.1
- Unified Manager 6.1
- Workflow Automation 2.2
- Performance Manager 1.0
NetApp today launched the EF550, an all-flash array delivering 450,000 sustained IOPS with sub-millisecond latency. This updated SANtricity-based system comes exactly nine months after the launch of the previous EF540.
The EF550 base models support 12 SSDs (9.6TB) or 24 SSDs (19.2TB), scaling up to 120 drives (96TB). As with the previous generation, the EF550 is a 2U form factor that is Storage Bridge Bay (SBB) version 2 compliant.
First, the controllers.
At the core of each EF550 controller is its quad-core Intel Xeon E5-2418L (“Sandy Bridge”) processor, which includes an integrated high-speed RAID engine to perform RAID 5 XOR and RAID 6 P + Q parity calculations with no performance penalty. This enables compute-intensive tasks to be handled both efficiently and effortlessly.
The controller’s PCI express Gen 3.0 x8 buses between the processor and external interfaces provide 24 gigabytes per second of internal bandwidth per controller, delivering both the “width” to handle large-block I/O and the “speed” to process large amounts of random, small-block I/O.
Each EF550 controller also provides 12GB of DDR3 SDRAM cache memory via three DIMMS. Its cache is battery-backed and destaged to internal flash upon power loss.
Next, let’s explore connectivity.
For connectivity, the EF550 supports various block protocols including:
- 16Gb FC (OM2, OM3, OS1, and OS2 optical)
- 10Gb iSCSI (optical, twinax passive, and RJ-45 Cat 6)
- 6Gb SAS (copper with Mini-SAS cables)
- 40Gb IB (QSFP+ copper and OM3 optical)
When attaching the EF550 directly to a server, or to switches in a SAN fabric, I/O paths to each controller should be established by having one connection to controller A and the other connection to controller B. This redundancy ensures data availability in the event of a path failure.