Migrate Permissions to Lake Formation
In this section we will migrate the Glue permissions to LakeFormation permissions.
Make sure to replace placeholder [AccountID] with the appropriate value before running the commands throughout this lab.
Step 1: List Users' and Roles' Existing Permissions
- Login as lf-admin
- Use Cloud9 to run the following command for both the users(glue-admin and glue-dev-user), whose permissions are being upgraded.
aws iam list-policies-granting-service-access --arn arn:aws:iam::[AccountID]:user/glue-admin --service-namespaces glue
- Using the information collected in previous step, grant AWS Lake Formation permissions to match the AWS Glue permissions in policy GlueProdPolicy and GlueTestPolicy
Step 2: Set Up Equivalent Lake Formation Permissions
- In the navigation pane, choose Data permissions.
- Choose Grant.
- In the Grant permissions dialog box, provide the following information:
- For IAM users and roles, choose glue-admin
- For Database, choose the database on which to grant permissions. Example: amazonprod
- For Database permissions, select the permissions that you want to grant.
- For Table permissions, select the table name.
- (Optional) For Grantable permissions, don’t do anything
- Follow same process to grant permissions to glue-dev-user
Step 3: Give Users IAM Permissions to Use Lake Formation
Now, give the two users permissions to use Lake Formation via the IAM policies created. Attach LakeFormationDataAccess
policy to the two users: glue-admin and glue-dev-user
. At this point in workshop, user policy list will look somewhat like the image below:
Step 4: Switch Your Data Stores to the Lake Formation Permissions Model
- Now, we will make switch to lake formation access model. It is recommended to upgrade to new model, one S3 bucket location at time. Repeat the process for each bucket that are referenced by your data catalog. Before registering a location, perform a verification step to ensure that the correct principals have the required Lake Formation permissions. For this, log in as admin to Cloud9 console you created. We will need to create an admin profile so will have to disable AWS managed temporary credentials as in image below. Use AWS Cloud9-->Preferences-->AWS Settings
- Go to IAM service and find credentials for admin user that you will need to create admin profile. In our case user is lf-admin. Run commands below :
aws configure --profile admin
- Enter information captured from previous steps and use region as us-east-1 and output format as json
aws lakeformation get-effective-permissions-for-path --resource-arn arn:aws:s3:::lf-data-lake-bucket-glue-lf-migration-[AccountID] --profile admin
- Output will be a json file that gives list of IAM principles with access to that S3 location.Now revoke ""Super" permission from IAMAllowedPrincipals on each table and database that you identified for the location
- In the navigation pane, choose Tables
- On the Tables page, select the radio button next to the desired table, amazon_reviews_test
- On the Actions menu, choose Revoke.
- In the Revoke permissionsdialog box, in the IAM users and roles list, scroll down to the Group heading, and choose IAMAllowedPrincipals.
- Under Table permissions, ensure that Superis selected, and then choose Revoke.
- Do same for test table(amazon_reviews_test)
- Now, let us do for databases. In the navigation pane, choose Databases.
- On the Databases page, select the radio button next to the desired database(amazonprod)
- On the Actions menu, choose revoke permissions as below
- Do same for amazontest database
- On the Edit databasepage, clear Use only IAM access control for new tables in this database, and then choose Save. Do same for amazontest database
- Now, register the Amazon S3 location with Lake Formation.
- In the navigation pane, under Register and Ingest, choose Data lake locations.
- Choose Register location, and then choose Browse to select an Amazon Simple Storage Service (Amazon S3) path.
- For IAM role, choose AWSServiceRoleForLakeFormationDataAccess
- Choose Register location.
Step 5: Secure New Data Catalog Resources
Next, secure all new Data Catalog resources by changing the default DataCatalog:
- In the navigation pane, choose Settings.
- On the Data catalog settingspage, clear both check boxes, and then choose Save.
Step 6: Give Users a New IAM Policy for Future Data Lake Access
Grant users access to additional databases or tables in the future by adding policy GlueFullReadAccess
with each user. It will look somewhat like the image below:
Step 7: Clean Up Existing IAM Policies
- As last step, clean up existing IAM policies.
- Remove GlueProdPolicy from glue-admin and GlueTestPolicy from glue-dev-user
- Remove Bucket Policies permission for glue-admin and glue-dev-user
- Log In as glue-admin and run Athena query. Do same for glue-dev-user. You will notice that even after removing bucket policies and glue catalogue permissions, user can run Athena. It is because LakeFormation is controlling the access now. You have successfully migrated from Glue to LakeFormation security model.
- Final state of glue-admin and glue-dev-user will be similar to image below: