Azure Data lake Gen 2 Suggestions


Ø  For blob storage, you organize a set of files/blobs under a container. In the Azure portal this is located in “Containers” under “Blob service”. It is called “Blob Containers” in both the portal and desktop Storage Explorer. For ADLS Gen2, you also use containers, located in the portal in “Containers” under “Data Lake Storage”. It is called “Blob Containers” in the desktop Storage Explorer but it is called “File Systems” in the portal Storage Explorer
Ø  In the Azure portal, for blob storage, you can upload/access files by going to the storage account and choosing Containers (under “Blob service”) or by using the Storage Explorer (preview) in the portal.  For ADLS Gen2, Containers (under “Data Lake Storage”) has no functionality except to create a container (click “File system”), but you can use the portal Storage Explorer. However, that is limited (you can’t upload files or change access tiers), so you should use the desktop Azure Storage Explorer to upload files or change access tiers
Ø  There are two types of storage performance tiers: Premium and Standard. The Premium performance tier can’t be changed to the Standard performance tier and visa-versa, so this is locked in when you create the storage account
Ø  The Premium performance tier is not yet available for ADLS Gen2, and only supports locally redundant storage (LRS)
Ø  There are three types of storage access tiers: Hot, Cool, and Archive. You can change access tiers with the Standard performance tier, but not with the Premium performance tier. Only block blobs support access tiers
Ø  When creating a storage account, you will be asked for the Account kind and you should use the default of General purpose v2 (StorageV2) unless you want to create block blobs or append blobs with the premium performance tier in which case you should choose Block Blob (BlockBlobStorage). Note that BlockBlobStorage accounts don’t currently support tiering to hot, cool, or archive access tiers
Ø  A storage access tier can be set for each file, but if it is not set it will default to the access tier (Hot or Cool) that the storage account is set to. The account access tier is the default tier that is inferred by any file without an explicitly set tier. The Archive access tier can only be set at the file level and not on the account
Ø  For blob storage, you can specify the access tier for a file (hot, cool, or archive) when uploading via the portal, but not when using the desktop Storage Explorer. For ADLS Gen2, there is not a way to upload files via the portal and you also can’t specify the access tier when uploading via the desktop Storage Explorer
Ø  For blob storage, to change an access tier, in the Azure portal, under the storage account, go to the container and choose the file and click “Change tier” to change its access tier. Or go to the portal or go to desktop Storage Explorer and right-click the file and choose “Change Access Tier”. For ADLS Gen2, you must use desktop Storage Explorer to change the access tier
Ø  For blob storage, you can specify that the blob type of a file is block, page, or append when uploading the file via the portal or with desktop Storage Explorer. Once a file has been created, its blob type cannot be changed. ADLS Gen2 only supports block blob type
Ø  Data in the Archive tier blob cannot be read until it is rehydrated to the Cool or Hot tier. The “standard” rehydration process can take up to 15 hours to complete.  There is a priority retrieval (called “high”) that takes less than an hour (see Azure Archive Storage expanded capabilities: faster, simpler, better).  You specify the rehydrate priority (standard or high) when choosing to switch from the archive tier on the portal. The option to choose the high priority retrieval is not available in the desktop Storage Explorer and is not available anywhere for ADLS Gen2

Comments

Post a Comment

Popular posts from this blog

Data Center Technician / Engineer certifications

Cisco Certified Network Associate Data Center (CCNA Data Center) Cisco Certified Network Professional Data Center ( CCNP Data Center ) Juniper Networks Certified Professional Data Center (JNCIP-DC) VCE Converged Infrastructure Administration Engineer (VCE-CIAE) VMware Certified Professional 6 – Data Center Virtualization (VCP6-DCV)
Read more

different OPB tables

OPB_SUBJECT - PowerCenter folders table OPB_MAPPING - Mappings table OPB_TASK - Tasks table like sessions, workflow etc TASK_TYPE for session is 68 and that of the workflow is 71. OPB_SESSION - Session & Mapping linkage table OPB_TASK_ATT R - Task attributes tables OPB_WIDGET - Transformations table Usage: Use WIDGET_ID from this table to that of the WIDGET_ID of any of the tables to know the transformation name and the folder details. Use this table in conjunction with OPB_WIDGET_ATTR or OPB_WIDGET_EXPR to know more about each transformation etc. OPB_WIDGET_FIEL D - Transformation ports table Usage: Take the FIELD_ID from this table and match it against the FIELD_ID of any of the tables like OPB_WIDGET_DEP and you can get the corresponding information. OPB_WIDGET_ATTR - Transformation properties table Usage: Use the ATTR_ID of this table to that of the ATTR_ID of OPB_ATTR table to find what each attribute in this transformation means. OPB_EXPRESSION - Expressions table Usa...
Read more

Industry 4.0 IOT

The Industrial Internet of Things (iIoT), which is also known as  Industry 4.0  when applied to the manufacturing industry, is a concept of integrating smart manufacturing machinery, AI-powered automation, and advanced analytics to help make every worker and every factory more efficient. The iIoT will revolutionize manufacturing by enabling the acquisition and accessibility of far greater amounts of data both faster and more efficiently than ever before. Many manufacturers have already started to implement the IIoT devices and processes by leveraging intelligent, connected devices inside their factories, warehouses, and workshops. In most cases, companies are deploying iIoT devices and machinery that are connected by communication technologies that assist industries in collecting, monitoring, analyzing, and delivering valuable insights like never before. The exact mechanics of how this technology is deployed varies from company to company, but the goal is always the same; to i...
Read more