diff --git a/README.md b/README.md index e87aaa9..d3edbc5 100644 --- a/README.md +++ b/README.md @@ -4,3 +4,4 @@ GakuNin RDM developer guide * [開発環境の準備](Environment.md) * [スケルトンの作成](Skelton/README.md) +* [ストレージアドオンの作成](StorageAddon/README.md) diff --git a/StorageAddon/README.md b/StorageAddon/README.md new file mode 100644 index 0000000..8f52bc3 --- /dev/null +++ b/StorageAddon/README.md @@ -0,0 +1,506 @@ +# ストレージアドオンの作成 + +具体的なストレージアドオンの例として、AWS S3互換の[MinIO](https://min.io/)と接続するストレージアドオンを実装します。このアドオンは特定のMinIOサーバーと接続するものとし、 My MinIOアドオンと名付けることとします。 + +基本的なアドオンの概要は[スケルトンの作成](../Skelton/README.md)を参照してください。 + +## 前提条件 + +[開発環境の準備](../Environment.md#開発環境でRDMを起動する)のガイドに従い、開発環境にてRDMを起動しているものとします。 + +# ストレージアドオンの設計 + +## サービスの構成 + +ストレージアドオンは以下の2つの要素から構成されます。 + +- OSF.ioサービスで動作するAddon: ユーザ・プロジェクト設定の管理 +- WaterButlerサービスで動作するProvider: ストレージへのアクセスの仲介 + +Addonによりユーザからの認証情報の受領や各種設定を行い、Providerはこの認証情報・設定情報をAddonから譲渡してもらい、実際のストレージへのアクセスを行います。 + +## ファイルの構成 + +典型的なクラス構成とファイル配置は以下のようになります。 + +### OSF.io Addonのクラス構成とファイル構成 + +![OSF.io Addon クラス構成](images/osf_class.png) + +`(*)`が付いているファイルはスケルトン アドオンには存在しないファイルです。 + +``` +/addons/アドオン名/ +├── __init__.py ... モジュールの定義 +├── apps.py ... アプリケーションの定義 +├── models.py ... モデルの定義 +├── provider.py ... (*) 認証プロバイダの定義 +├── requirements.txt ... 利用するPythonモジュールの定義 +├── routes.py ... View(Routes)の定義 +├── serializer.py ... (*) モデル-ビュー(JavaScript)間の情報交換用シリアライザの定義 +├── settings ... 設定を定義するモジュール +│ ├── defaults.py ... デフォルト設定の定義 +│ ├── __init__.py ... 設定の定義 +│   └── local-dist.py ... (*) local.pyのサンプルファイル +├── static ... Webブラウザから読み込むことを想定した静的ファイル +│   ├── comicon.png ... アドオンのアイコン +│   ├── myminioAnonymousLogActionList.json ... (*) 変更履歴メッセージ定義 +│   ├── myminioLogActionList.json ... (*) 変更履歴メッセージ定義 +│   ├── myminioNodeConfig.js ... (*) Node設定の定義 +│   ├── myminioUserConfig.js ... (*) User設定の定義 +│   ├── node-cfg.js ... Node設定のエントリとなるJavaScriptファイル +│   └── user-cfg.js ... (*) User設定のエントリとなるJavaScriptファイル +├── templates ... テンプレートディレクトリ +│   ├── credentials_modal.mako ... (*) 認証情報の設定用ダイアログ +│   ├── node_settings.mako ... Node設定パネル +│   └── user_settings.mako ... (*) User設定パネル +├── tests ... テストコード +│   ├── __init__.py +│   ├── conftest.py +│   ├── factories.py +│   ├── test_model.py +│   ├── test_serializer.py ... (*) +│   ├── test_view.py +│   └── utils.py +├── utils.py ... (*) ユーティリティ関数の定義 +└── views.py ... View(Views)の定義 +``` + +### WaterButler Providerのクラス構成とファイル構成 + +![WaterButler Addon クラス構成](images/wb_class.png) + +``` +waterbutler/providers/アドオン名/ +├── __init__.py ... Providerクラスの参照 +├── metadata.py ... Metadataの定義 +├── provider.py ... Providerの定義 +└── settings.py ... デフォルト設定の定義 + +tests/providers/アドオン名/ +├── __init__.py +└── provider.py ... Providerのテストコード +``` + +## OSF.io Addonのモジュール構成 + +スケルトン アドオンとの違いを中心に説明していきます。 + +### Modelの構成 + +`models.py` に、以下のModelを定義します。 + +- `UserSettings`: ユーザーに関する情報(認証情報等) +- `NodeSettings`: プロジェクトに関する情報 +- `アドオン名FileNode`: ファイル・フォルダオブジェクトの親定義 +- `アドオン名File`: ファイルオブジェクトの定義 +- `アドオン名Folder`: ファイルオブジェクトの定義 + +`provider.py` に `アドオン名Provider` を、 `serializer.py` に `アドオン名Serializer` を定義します。OAuthを用いる場合は `osf.models.external.ExternalProvider` を継承し、必要なメンバを定義します。実装は [GitHubの例](https://github.com/RCOSDP/RDM-osf.io/blob/develop/addons/github/models.py#L49) などを参考にしてください(GitHubアドオンは `アドオン名Provider` を `models.py` に定義しています)。 +`アドオン名Serializer` にはビュー(JavaScript)にURLリストや接続先フォルダなどを渡すための関数を定義します。 + +OAuth認証をしない場合であっても、ストレージアドオンの `UserSettings` と `NodeSettings` は、 `BaseOAuthUserSettings` と `BaseOAuthNodeSettings` をそれぞれ継承して定義し、 `oauth_provider` に`アドオン名Provider` を指定します。こうすることで、統一的な構造で簡単に認証の仕組みを実装することができます。 + +My MinIOアドオンの場合は、以下のように定義します。 + +- [models.py](osf.io/addon/models.py) +- [provider.py](osf.io/addon/provider.py) +- [serializer.py](osf.io/addon/serializer.py) + +### Viewの構成 + +`views.py`には、おおよそ以下の関数を定義します。 + +| 関数名 | 処理 | +|:------|:----| +| set_config | プロジェクトの設定を保存する。デフォルトでは接続先のフォルダとクライアント用のAPIのURLリストのみ。 | +| get_config | プロジェクトの設定を取得する。 | +| import_auth | ログイン中のユーザの認証情報をプロジェクトにインポートする。 | +| deauthorize_node | プロジェクトの認証情報を取り消す。 | +| add_user_account | ユーザの認証情報を追加する。 | +| account_list | ユーザの認証情報リストを取得する。 | +| create_folder | ストレージサービスにフォルダを追加する。プロジェクトのアドオン設定ページでフォルダを作る機能を提供しない場合は不要。 | +| folder_list | ストレージサービスのフォルダリストを取得する。 | + +シリアライザ(`serializer.py`)を定義し、 `addons.base.generic_views` を利用することで、以下のように一部のView処理を簡単に定義することができます。 + +``` +import_auth = addons.base.generic_views.import_auth( + SHORT_NAME, + Serializer +) +``` + +`generic_views` で提供していないView処理を追加したい場合や、View処理をカスタマイズしたい場合は、以下のように `views.py` に個別の関数を定義します。 + +``` +@must_have_addon(SHORT_NAME, 'node') +@must_be_addon_authorizer(SHORT_NAME) +def folder_list(node_addon, **kwargs): + return node_addon.get_folders() +``` + +フォルダの作成やフォルダリストの取得をするために、Addonでもストレージサービスへ接続する必要があります。 + +My MinIOアドオンの場合は、以下のように定義します。 + +- [routes.py](osf.io/addon/routes.py) +- [views.py](osf.io/addon/views.py) + + +### フレームワークによって提供されるView + +アドオンが持つ利用者用設定画面(`user_settings.mako`)とプロジェクト用設定画面(`node_settings.mako`)のテンプレートをそれぞれ定義します。認証情報の設定用ダイアログ(`credentials_modal.mako`)はどちらの画面でも利用するので、別のファイルで定義し、それぞれから参照します。 +My MinIOアドオンの場合は、以下のように定義します。 + +- [user_settings.mako](osf.io/addon/templates/user_settings.mako) +- [node_settings.mako](osf.io/addon/templates/node_settings.mako) +- [credentials_modal.mako](osf.io/addon/templates/credentials_modal.mako) + +利用者用設定画面のJavaScriptファイル(`user-cfg.js`, `myminioUserConfig.js`)と、プロジェクト用設定画面のJavaScriptファイル(`node-cfg.js`, `myminioNodeConfig.js`)をそれぞれ定義します。今回は、エントリとなるJavaScriptファイル(`*-cfg.js`)と定義ファイル(`myminio*Config.js`)を分けましたが、スケルトン アドオンのように `*-cfg.js` に定義を書いても構いません。 +My MinIOアドオンの場合は、以下のように定義します。 + +- [user-cfg.js](osf.io/addon/static/user-cfg.js) +- [node-cfg.js](osf.io/addon/static/node-cfg.js) +- [myminioUserConfig.js](osf.io/addon/static/myminioUserConfig.js) +- [myminioNodeConfig.js](osf.io/addon/static/myminioNodeConfig.js) + +また、変更履歴メッセージの定義ファイルも追加します。 +My MinIOアドオンの場合は、以下のように定義します。 + +- [myminioAnonymousLogActionList.json](osf.io/addon/static/myminioAnonymousLogActionList.json) +- [myminioLogActionList.json](osf.io/addon/static/myminioLogActionList.json) + +ストレージ操作UIであるFileViewerは[Fangorn](https://github.com/RCOSDP/RDM-osf.io/blob/develop/website/static/js/fangorn.js)を使って実装されています。アイテム選択時に表示するボタンをカスタマイズしたい場合は、 `Fangorn.config.アドオン名` を定義し、 `files.js` ファイルで読み込みます。 +My MinIOアドオンではカスタマイズせずデフォルト動作を使用しています。カスタマイズ例は、[GitHubアドオン](https://github.com/RCOSDP/RDM-osf.io/blob/develop/addons/github/static/githubFangornConfig.js)や[IQB-RIMSアドオン](https://github.com/RCOSDP/RDM-osf.io/blob/develop/addons/iqbrims/static/iqbrimsFangornConfig.js)を参照してください。 + +### 設定モジュール + +環境ごとの設定ファイル `local.py` の雛形として `local-dist.py` ファイルを定義します。サービス管理者は `local-dist.py` を `local.py` にコピーして、適宜設定値を書き換えます。 + +My MinIOアドオンの場合は、以下のように定義します。接続するMinIOサービスのホスト名を `HOST` プロパティに設定します。 + +- [local-dist.py](osf.io/addon/settings/local-dist.py) + +### テストコード + +シリアライザのテスト(`test_serializer.py`)を定義します。 +My MinIOアドオンの場合は、以下のように定義します。 + +- [test_serializer.py](osf.io/addon/tests/test_serializer.py) + +## WaterButler Providerのモジュール構成 + +WaterButler Providerは、WaterButlerプロジェクトの中でPythonモジュールとして実装されます。 + +`metadata.py` には、フォルダやファイルのメタデータと、リビジョンを持つクラスを定義します。 +My MinIOアドオンの場合は、以下のように定義します。 + +- [metadata.py](waterbutler/provider/metadata.py) + +`provider.py`には、ストレージサービスと接続しCRUD操作などを行うProviderクラスを定義します。 + +Providerが提供するメソッドには以下のようなものがあります。 + + +| 関数名 | 引数 | 戻り値 | 処理 | +|:------|:----|:------|:-----| +| validate_v1_path | path, **kwargs | WaterButlerPath | 文字列で与えられたパス情報(`path`)を検証し、属性付きのWaterButlerPathオブジェクトを返す。 | +| validate_path | path, **kwargs | WaterButlerPath | 同上(廃止予定のv0仕様との互換性維持のため、2つのメソッドに分かれている)。 | +| download | path, accept_url=False, version=None, range=None, **kwargs | Stream | 指定されたパス(`path`)のデータをダウンロードする。戻り値にはデータアクセス用のStreamを返す。 | +| upload | stream, path, conflict='replace', **kwargs | Metadata | 指定されたパス(`path`)に指定されたデータ(`stream`)をアップロードする。戻り値にはアップロードしたファイルを示すMetadataを返す。 | +| delete | path, confirm_delete=0, **kwargs | なし | 指定されたパス(`path`)のファイル・フォルダを削除する。 | +| revisions | path, **kwargs | List(Revision) | 指定されたパス(`path`)のリビジョン情報を取得する。 | +| metadata | path, revision=None, **kwargs | Metadata or List(Metadata) | 指定されたパス(`path`)のメタデータを取得する。`path` がfileの場合 `Metadata` , directoryの場合 `List(Metadata)` を返す。 | +| create_folder | path, folder_precheck=True, **kwargs | Metadata | 指定されたパスにフォルダを作成する。戻り値には作成したフォルダを示すMetadataを返す。 | +| can_intra_copy | dest_provider, path=None | Bool | 指定された送信先Provider(`dest_provider`), パス(`path`)に対してintra_copy(内部コピー: ストレージサービス上でのコピー)が可能かどうかを判定する。これがFalseの場合、いったん一時ディレクトリにdownloadして、destにuploadするという操作となる。 | +| can_intra_move | dest_provider, path=None | Bool | 指定された送信先Provider(`dest_provider`), パス(`path`)に対してintra_move(内部移動: ストレージサービス中での移動)が可能かどうかを判定する。これがFalseの場合、いったん一時ディレクトリにdownloadして、destにupload、コピー元ファイルをdeleteするという操作となる。 | +| intra_copy | dest_provider, src_path, dest_path | Bool | 内部コピーを実施する。成功すればTrueを返す。 | +| intra_move | dest_provider, src_path, dest_path | Bool | 内部移動を実施する。成功すればTrueを返す。 | + +My MinIOアドオンの場合は、以下のように定義します。 + +- [provider.py](waterbutler/provider/provider.py) + + +## OSF.io AddonとWaterButler Providerの認証情報の委譲 + +OSF.io Addonは、WaterButler Providerが必要なサービスに接続できるよう、認証情報を委譲します。この認証情報の形式はAddonにより異なるため、AddonとProviderのバージョンを合わせるなど依存関係に配慮する必要があります。 + +![認証情報などの委譲](images/crud.png) + +Addonにおける委譲設定は、 `addons.アドオン名.models.NodeSettings.serialize_waterbutler_credentials()` 関数で設定します。接続先のフォルダなどの情報は、同クラスの `serialize_waterbutler_settings()` 関数で設定します。例えば [IQB-RIMSアドオンの `serialize_waterbutler_settings()` 関数](https://github.com/RCOSDP/RDM-osf.io/blob/develop/addons/iqbrims/models.py#L226) では、フォルダごとの権限設定を渡します。 + +My MinIOアドオンの場合は、以下のように定義します。認証情報として、アクセス先のホスト名、アクセスキー、シークレットキーを渡します。設定情報として、接続先のバケットIDを渡します。 + +``` +def serialize_waterbutler_credentials(self): + if not self.has_auth: + raise exceptions.AddonError('Cannot serialize credentials for {} addon'.format(FULL_NAME)) + return { + 'host': settings.HOST, + 'access_key': self.external_account.oauth_key, + 'secret_key': self.external_account.oauth_secret, + } + +def serialize_waterbutler_settings(self): + if not self.folder_id: + raise exceptions.AddonError('Cannot serialize settings for {} addon'.format(FULL_NAME)) + return { + 'bucket': self.folder_id + } +``` + +Provider側では、認証情報と設定情報を `waterbutler.providers.アドオン名.アドオン名Provider` クラスのコンストラクタ(`__init__()`)の引数 `credentials` と `settings` にDictionary型でそれぞれ受け取ります。 + +My MinIOアドオンの場合は、以下のように定義します。認証情報を使ってMy MinIOサービスとの接続を確立し、設定情報を使って接続先バケットを取得します。 + +``` +class MyMinIOProvider(provider.BaseProvider): + def __init__(self, auth, credentials, settings): + super().__init__(auth, credentials, settings) + + host = credentials['host'] + port = 443 + m = re.match(r'^(.+)\:([0-9]+)$', host) + if m is not None: + host = m.group(1) + port = int(m.group(2)) + self.connection = MyMinIOConnection(credentials['access_key'], + credentials['secret_key'], + calling_format=OrdinaryCallingFormat(), + host=host, + port=port, + is_secure=port == 443) + self.bucket = self.connection.get_bucket(settings['bucket'], validate=False) +``` + + +# My MinIOアドオンの実装 + +ここでは、 `myminio` という識別名のアドオンの実装を例に説明します。アドオンの完全名は `My MinIO` とします。 + +アドオン名は様々な場所に埋め込まれています。アドオン名を変更したい場合は、以降で追加・変更するファイル名やコードの `myminio` 、 `My MinIO` 、 `MyMinIO` という文字列を変更してください。 + +My MinIOアドオンは、AWS S3互換サービスと接続するアドオンなので、Amazon S3アドオンやS3 Compatible Storageアドオンを参考に実装することができます。Amazon S3アドオンやS3 Compatible Storageアドオンとの違いは以下の通りです。 + +- My MinIOアドオンは簡単のため、アップロードの暗号化機能を実装しない。 +- Amazon S3アドオンやS3 Compatible Storageアドオンは、アカウントごとに接続するサービスやLocationなどを選択できるが、My MinIOはサービス側で指定した特定のMinIOサービスのみを扱う。 + +## OSF.ioへの実装 + +### addons.myminio モジュールの定義 + +スケルトン アドオンと同様に、[アドオンの実装例(`osf.io/addon/`)](osf.io/addon/)を `addons/myminio` ディレクトリにコピーします。 + +### RDMコードの変更 + +スケルトン アドオンと同様に、RDMのコードをいくつか変更します。 + +- [addons.json](https://github.com/RCOSDP/RDM-osf.io/blob/develop/addons.json) + - 変更例のサンプル: [addons.json](osf.io/config/addons.json) +- [framework/addons/data/addons.json](https://github.com/RCOSDP/RDM-osf.io/blob/develop/framework/addons/data/addons.json) + - 変更例のサンプル: [addons.json](osf.io/config/framework/addons/data/addons.json) +- [Dockerfile](https://github.com/RCOSDP/RDM-osf.io/blob/develop/Dockerfile) + - 変更例のサンプル: [Dockerfile](osf.io/config/Dockerfile) +- [api/base/settings/defaults.py](https://github.com/RCOSDP/RDM-osf.io/blob/develop/api/base/settings/defaults.py) + - 変更例のサンプル: [defaults.py](osf.io/config/api/base/settings/defaults.py) + +`api/base/settings/defaults.py` は、 `INSTALLED_APPS` の他に、 `ADDONS_FOLDER_CONFIGURABLE` 、 `ADDONS_OAUTH` にもアドオン名を追加します。変更例サンプルでは、Amazon S3アドオンに合わせて、スケルトン アドオンとは異なる方法で設定しています。 + +``` +INSTALLED_APPS += ('addons.myminio',) +ADDONS_FOLDER_CONFIGURABLE.append('myminio') +ADDONS_OAUTH.append('myminio') +``` + +他にもストレージアドオンでは、 [api/base/settings/defaults.py](https://github.com/RCOSDP/RDM-osf.io/blob/develop/api/base/settings/defaults.py) にも設定を追加する必要があります。 + +変更例は [storageAddons.json](osf.io/config/website/static/storageAddons.json) を参照してください。 + +``` + "myminio": { + "fullName": "My MinIO", + "externalView": false + }, +``` + +> `externalView` を `true` に設定すると、FileViewerでファイルの外部ページリンクボタンが表示されるようになります。リンクを正しく動作させるには、WaterButlerのProviderを修正する必要があります。詳細は[GoogleDriveの実装](https://github.com/RCOSDP/RDM-waterbutler/blob/develop/waterbutler/providers/googledrive/metadata.py#L116)などを参照してください。 + +> FileViewerで、フォルダの操作はできるけどファイルの操作ができない場合は、 `storageAddons.json` の設定が漏れている可能性があります。 + + +### Migrationsファイルの作成 + +`makemigrations` コマンドを実行して、Migrationsファイルを作成します。 + +``` +$ docker-compose run --rm web python3 manage.py makemigrations +``` + +上記の出力中に以下のような出力が現れれば成功です。ストレージアドオンの場合、 `osf/migrations` と `addons/myminio/migrations` の2つのディレクトリ内にPythonファイルが作成されます。 + +> `osf/migrations` 配下に作成されるファイル名は、作成日時やRDMのバージョンによって異なります。 + +``` +Migrations for 'osf': + osf/migrations/0214_auto_20201001_0007.py + - Create proxy model MyMinIOFileNode + - Alter field type on basefilenode + - Create proxy model MyMinIOFile + - Create proxy model MyMinIOFolder +Migrations for 'addons_myminio': + addons/myminio/migrations/0001_initial.py + - Create model NodeSettings + - Create model UserSettings + - Add field user_settings to nodesettings +``` + +### アドオンのテスト + +以下のコマンドで、OSF.ioに追加したMy MinIOアドオンのユニットテストを実行できます。 + +``` +$ docker-compose run --rm web invoke test_module -m addons/myminio/tests/ +``` + +## WaterButlerへの実装 + +### waterbutler.providers.myminio モジュールの定義 + +[Providerの実装例(`waterbutler/provider/`)](waterbutler/provider/) を `waterbutler/providers/myminio` にコピーします。また、[Providerのテストコード例(`waterbutler/tests`)](waterbutler/tests) を `tests/providers/myminio` にコピーします。 + +### RDMコードの変更 + +[setup.py](https://github.com/RCOSDP/RDM-waterbutler/blob/develop/setup.py) に、アドオンのエントリポイント定義を追加します。`setup()` 関数の引数 `entry_points` に指定するDictionaryの `waterbutler.providers` キーに指定する配列に、以下を追加します。 + +``` +'myminio = waterbutler.providers.myminio:MyMinIOProvider', +``` + +変更例はサンプル [setup.py](waterbutler/config/setup.py) を参照してください。 + +### docker-compose.override.yml の追加 + +変更したコードを使うよう、RDMの `docker-compose.override.yml` で WaterButler のサービスの `volumes` を指定します。 + +例えば、以下のようなディレクトリ階層の場合は、 + +``` +. +├── RDM-osf.io +│   ├── docker-compose.override.yml +│   └── docker-compose.yml +└── RDM-waterbutler +``` + +`RDM-osf.io/docker-compose.override.yml` の内容を以下のようにします。 + +``` +version: "3.4" + +services: + wb: + volumes: + - ../RDM-waterbutler:/code + wb_worker: + volumes: + - ../RDM-waterbutler:/code + wb_requirements: + volumes: + - ../RDM-waterbutler:/code +``` + + +### アドオンのテスト + +以下のコマンドで、WaterButlerに追加したMy MinIOアドオン Providerのユニットテストを実行できます。 + +``` +$ docker-compose run --rm wb invoke test --provider myminio +``` + + +# My MinIOアドオンの動作確認 + +My MinIOアドオンの動作確認をしてみましょう。 + +## MinIOサービスの起動 + +接続先のMinIOサービスを起動します。 + +``` +docker run -p 9001:9000 \ + -e "MINIO_ACCESS_KEY=minioadmin" \ + -e "MINIO_SECRET_KEY=minioadmin" \ + minio/minio server /data +``` + +RDMと同じコンピュータで実行する場合、9000番ポートが競合してしまうので、MinIOサービスのホストへの割り当てポートは9001などに設定します。 +`MINIO_ACCESS_KEY` と `MINIO_SECRET_KEY` は認証に使うキーです。適宜書き換えてください。 + +その他MinIOに関する詳しい説明は[MinIOのドキュメント](https://docs.min.io/)を参照してください。 + +## OSF.io Addonの設定 + +OSF.io Addonの設定をします。 `addons/myminio/settings/local-dist.py` を `addons/myminio/settings/local.py` にコピーし、 `HOST` プロパティに先ほど起動したMinIOサービスのホスト名を指定します。 + +``` +HOST = '192.168.168.167:9001' +``` + +RDMとMinIOサービスを同じ環境で実行している場合は、ループバックエイリアスを指定します。 + +## DBマイグレーション + +マイグレーションを実行し、Migrations定義をPostgreSQLサービスに反映します。 + +``` +$ docker-compose run --rm web python3 manage.py migrate +``` + +## サービスの再起動 + +WaterButlerに追加したProviderを有効にするために、 `wb_requirements` を起動します。 + +``` +$ docker-compose up wb_requirements +``` + +変更したファイルに関連するサービスを再起動します。 + +``` +$ docker-compose restart assets web api wb +``` + +これでサービスへの反映は完了です。 + +## ストレージアドオンを試す + +My MinIOアドオンを試すには、以下のような操作を実施します。 + +1. RDM Web UIにアクセスする `http://localhost:5000` +1. ユーザ設定ページを開く +1. Configure add-on accountsページを開く +1. My MinIOアドオンの認証情報を設定をする + ![Authorize Addon](images/authorize-addon.png) + 認証情報の設定ダイアログが表示されるので、MinIOサービスの `MINIO_ACCESS_KEY` と `MINIO_SECRET_KEY` を、それぞれ、`Access Key` と `Secret Key` フォームに入力して保存します。 + ![Filled Credential](images/filled-credential.png) + 成功すれば、切断ボタンと利用しているプロジェクトリストが表示されるエリアが表示されます。 + ![Successful Authorization Addon](images/successful-authorization-addon.png) +1. 適当なプロジェクトを作成する +1. Add-onsページを開く +1. My MinIOアドオンを有効化する + ![Enable Addon](images/enable-addon.png) +1. My MinIOアドオンの設定をする + ![Configure Addon](images/configure-addon.png) + `Import Account from Profile`ボタンから、アドオンの設定を行います。 + 接続に成功すると、ルート直下のフォルダリストが表示されるので、プロジェクトに紐付けるフォルダを選択し、保存します。このページでフォルダを作成することもできます。 + ![Select Current Folder](images/select-current-folder.png) + +これで、作成したプロジェクトでMy MinIOアドオンが使えるようになりました。 +プロジェクトページのFiles ウィジェットやFilesページから、My MinIOサービスに対してフォルダの作成やファイルのアップロード、削除、ダウンロードなどができるようになるはずです。 + +![Enabled Addon](images/enabled-addon.png) + +以上でMy MinIOアドオンの動作確認は完了です! diff --git a/StorageAddon/images/authorize-addon.png b/StorageAddon/images/authorize-addon.png new file mode 100644 index 0000000..952cd79 Binary files /dev/null and b/StorageAddon/images/authorize-addon.png differ diff --git a/StorageAddon/images/configure-addon.png b/StorageAddon/images/configure-addon.png new file mode 100644 index 0000000..23b79d7 Binary files /dev/null and b/StorageAddon/images/configure-addon.png differ diff --git a/StorageAddon/images/crud.png b/StorageAddon/images/crud.png new file mode 100644 index 0000000..ecf4783 Binary files /dev/null and b/StorageAddon/images/crud.png differ diff --git a/StorageAddon/images/crud.uml b/StorageAddon/images/crud.uml new file mode 100644 index 0000000..f41ad66 --- /dev/null +++ b/StorageAddon/images/crud.uml @@ -0,0 +1,23 @@ +# Build: +# $ cat crud.uml | docker run --rm -i think/plantuml -tpng > crud.png +@startuml + +actor User as user +participant "OSF.io Addon" as osf +participant "WaterButler Provider" as wb +database "External Storage" as storage + +osf --> user : Cookie +user -> wb : CRUD Request (+ Cookie) +activate wb +'wb -> osf : Request +'activate osf +'return Credentials and Settings +osf --> wb : Credentials and Settings +wb -> storage : CRUD Request +activate storage +return (Files) +return (Files) + + +@enduml diff --git a/StorageAddon/images/enable-addon.png b/StorageAddon/images/enable-addon.png new file mode 100644 index 0000000..a03e97c Binary files /dev/null and b/StorageAddon/images/enable-addon.png differ diff --git a/StorageAddon/images/enabled-addon.png b/StorageAddon/images/enabled-addon.png new file mode 100644 index 0000000..b4776bb Binary files /dev/null and b/StorageAddon/images/enabled-addon.png differ diff --git a/StorageAddon/images/filled-credential.png b/StorageAddon/images/filled-credential.png new file mode 100644 index 0000000..4ce59bc Binary files /dev/null and b/StorageAddon/images/filled-credential.png differ diff --git a/StorageAddon/images/osf_class.png b/StorageAddon/images/osf_class.png new file mode 100644 index 0000000..34d7af8 Binary files /dev/null and b/StorageAddon/images/osf_class.png differ diff --git a/StorageAddon/images/osf_class.uml b/StorageAddon/images/osf_class.uml new file mode 100644 index 0000000..27a08c3 --- /dev/null +++ b/StorageAddon/images/osf_class.uml @@ -0,0 +1,96 @@ +# Build: +# $ cat osf_class.uml | docker run --rm -i think/plantuml -tpng > osf_class.png +@startuml + +package addons.base.apps <> { + class BaseAddonAppConfig +} + +package addons.base <> { + object generic_views +} + +package addons.base.serializer <> { + class StorageAddonSerializer +} + +package addons.base.models <> { + class BaseNodeSettings + class BaseUserSettings + class BaseOAuthNodeSettings + class BaseOAuthUserSettings + class BaseStorageAddon +} + +package osf.models.files <> { + class BaseFileNode + class Folder + class File +} + + +package addons.$(addon_short_name).apps <> { + class AddonAppConfig +} + +package addons.$(addon_short_name).routes <> { + object api_routes +} + +package addons.$(addon_short_name).provider <> { + class "$(AddonShortName)Provider" +} + +package addons.$(addon_short_name).serializer <> { + class "$(AddonShortName)Serializer" +} + +package addons.$(addon_short_name).models <> { + class NodeSettings + class UserSettings + class "$(AddonShortName)FileNode" + class "$(AddonShortName)Folder" + class "$(AddonShortName)File" +} + +package addons.$(addon_short_name).views <> { + object set_config + object get_config + object import_auth + object deauthorize_node + object add_user_account + object account_list + object create_folder + object folder_list +} + +BaseAddonAppConfig <|-- AddonAppConfig + +StorageAddonSerializer <|-- "$(AddonShortName)Serializer" + +BaseNodeSettings <|-- BaseOAuthNodeSettings +BaseUserSettings <|-- BaseOAuthUserSettings + +BaseOAuthNodeSettings <|-- NodeSettings +BaseStorageAddon <|-- NodeSettings +NodeSettings ..> "$(AddonShortName)Provider" +NodeSettings ..> "$(AddonShortName)Serializer" + +BaseOAuthUserSettings <|-- UserSettings +UserSettings ...> "$(AddonShortName)Provider" +UserSettings ...> "$(AddonShortName)Serializer" + +BaseFileNode <|-- "$(AddonShortName)FileNode" +Folder <|-- "$(AddonShortName)Folder" +File <|-- "$(AddonShortName)File" +"$(AddonShortName)FileNode" <|-- "$(AddonShortName)Folder" +"$(AddonShortName)FileNode" <|-- "$(AddonShortName)File" + +AddonAppConfig ..> api_routes +AddonAppConfig ..> NodeSettings +AddonAppConfig ..> UserSettings +api_routes ..> "addons.$(addon_short_name).views" +"addons.$(addon_short_name).views" ..> generic_views +"addons.$(addon_short_name).views" ..> "$(AddonShortName)Serializer" + +@enduml diff --git a/StorageAddon/images/select-current-folder.png b/StorageAddon/images/select-current-folder.png new file mode 100644 index 0000000..f28526f Binary files /dev/null and b/StorageAddon/images/select-current-folder.png differ diff --git a/StorageAddon/images/successful-authorization-addon.png b/StorageAddon/images/successful-authorization-addon.png new file mode 100644 index 0000000..b87f043 Binary files /dev/null and b/StorageAddon/images/successful-authorization-addon.png differ diff --git a/StorageAddon/images/wb_class.png b/StorageAddon/images/wb_class.png new file mode 100644 index 0000000..235092b Binary files /dev/null and b/StorageAddon/images/wb_class.png differ diff --git a/StorageAddon/images/wb_class.uml b/StorageAddon/images/wb_class.uml new file mode 100644 index 0000000..3888be6 --- /dev/null +++ b/StorageAddon/images/wb_class.uml @@ -0,0 +1,45 @@ +# Build: +# $ cat wb_class.uml | docker run --rm -i think/plantuml -tpng > wb_class.png +@startuml + +package waterbutler.core.metadata <> { + class BaseMetadata + class BaseFileMetadata + class BaseFolderMetadata + class BaseFileRevisionMetadata +} + +package waterbutler.core.provider <> { + class BaseProvider +} + + +package waterbutler.providers.$(addon_short_name).metadata <> { + class "$(AddonShortName)Metadata" + class "$(AddonShortName)FileMetadataHeaders" + class "$(AddonShortName)FileMetadata" + class "$(AddonShortName)FolderKeyMetadata" + class "$(AddonShortName)FolderMetadata" + class "$(AddonShortName)Revision" +} + +package waterbutler.providers.$(addon_short_name).provider <> { + class "$(AddonShortName)Provider" +} + + +BaseMetadata <|-- "$(AddonShortName)Metadata" +"$(AddonShortName)Metadata" <|-- "$(AddonShortName)FileMetadataHeaders" +BaseFileMetadata <|-- "$(AddonShortName)FileMetadataHeaders" +"$(AddonShortName)Metadata" <|-- "$(AddonShortName)FileMetadata" +BaseFileMetadata <|-- "$(AddonShortName)FileMetadata" +"$(AddonShortName)Metadata" <|-- "$(AddonShortName)FolderKeyMetadata" +BaseFolderMetadata <|-- "$(AddonShortName)FolderKeyMetadata" +"$(AddonShortName)Metadata" <|-- "$(AddonShortName)FolderMetadata" +BaseFolderMetadata <|-- "$(AddonShortName)FolderMetadata" +BaseFileRevisionMetadata <|-- "$(AddonShortName)Revision" + +BaseProvider <|-- "$(AddonShortName)Provider" +"$(AddonShortName)Provider" ..> "waterbutler.providers.$(addon_short_name).metadata" + +@enduml diff --git a/StorageAddon/osf.io/addon/__init__.py b/StorageAddon/osf.io/addon/__init__.py new file mode 100644 index 0000000..0bdb5eb --- /dev/null +++ b/StorageAddon/osf.io/addon/__init__.py @@ -0,0 +1,4 @@ +SHORT_NAME = 'myminio' +FULL_NAME = 'My MinIO' + +default_app_config = 'addons.{}.apps.MyMinIOAddonAppConfig'.format(SHORT_NAME) diff --git a/StorageAddon/osf.io/addon/apps.py b/StorageAddon/osf.io/addon/apps.py new file mode 100644 index 0000000..7b9dc03 --- /dev/null +++ b/StorageAddon/osf.io/addon/apps.py @@ -0,0 +1,70 @@ +import os + +from addons.base.apps import BaseAddonAppConfig, generic_root_folder +from addons.myminio import SHORT_NAME, FULL_NAME +from addons.myminio import settings + +myminio_root_folder = generic_root_folder('myminio') + +HERE = os.path.dirname(os.path.abspath(__file__)) +TEMPLATE_PATH = os.path.join( + HERE, + 'templates' +) + + +class MyMinIOAddonAppConfig(BaseAddonAppConfig): + + name = 'addons.{}'.format(SHORT_NAME) + label = 'addons_{}'.format(SHORT_NAME) + full_name = FULL_NAME + short_name = SHORT_NAME + + owners = ['user', 'node'] + configs = ['accounts', 'node'] + categories = ['storage'] + + has_hgrid_files = True + + max_file_size = settings.MAX_FILE_SIZE + + node_settings_template = os.path.join(TEMPLATE_PATH, 'node_settings.mako') + user_settings_template = os.path.join(TEMPLATE_PATH, 'user_settings.mako') + + BUCKET_LINKED = 'myminio_bucket_linked' + BUCKET_UNLINKED = 'myminio_bucket_unlinked' + FILE_ADDED = 'myminio_file_added' + FILE_REMOVED = 'myminio_file_removed' + FILE_UPDATED = 'myminio_file_updated' + FOLDER_CREATED = 'myminio_folder_created' + NODE_AUTHORIZED = 'myminio_node_authorized' + NODE_DEAUTHORIZED = 'myminio_node_deauthorized' + NODE_DEAUTHORIZED_NO_USER = 'myminio_node_deauthorized_no_user' + actions = ( + BUCKET_LINKED, + BUCKET_UNLINKED, + FILE_ADDED, + FILE_REMOVED, + FILE_UPDATED, + FOLDER_CREATED, + NODE_AUTHORIZED, + NODE_DEAUTHORIZED, + NODE_DEAUTHORIZED_NO_USER + ) + + @property + def get_hgrid_data(self): + return myminio_root_folder + + @property + def routes(self): + from . import routes + return [routes.api_routes] + + @property + def user_settings(self): + return self.get_model('UserSettings') + + @property + def node_settings(self): + return self.get_model('NodeSettings') diff --git a/StorageAddon/osf.io/addon/migrations/__init__.py b/StorageAddon/osf.io/addon/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/StorageAddon/osf.io/addon/models.py b/StorageAddon/osf.io/addon/models.py new file mode 100644 index 0000000..0a610c3 --- /dev/null +++ b/StorageAddon/osf.io/addon/models.py @@ -0,0 +1,146 @@ +from django.db import models + +import addons.myminio.settings as settings +from addons.base import exceptions +from addons.base.models import (BaseOAuthNodeSettings, BaseOAuthUserSettings, + BaseStorageAddon) +from addons.myminio import SHORT_NAME, FULL_NAME +from addons.myminio.provider import MyMinIOProvider +from addons.myminio.serializer import MyMinIOSerializer +from addons.myminio.utils import bucket_exists, get_bucket_names +from framework.auth.core import Auth +from osf.models.files import File, Folder, BaseFileNode + + +class MyMinIOFileNode(BaseFileNode): + _provider = SHORT_NAME + + +class MyMinIOFolder(MyMinIOFileNode, Folder): + pass + + +class MyMinIOFile(MyMinIOFileNode, File): + version_identifier = 'version' + + +class UserSettings(BaseOAuthUserSettings): + oauth_provider = MyMinIOProvider + serializer = MyMinIOSerializer + + +class NodeSettings(BaseOAuthNodeSettings, BaseStorageAddon): + oauth_provider = MyMinIOProvider + serializer = MyMinIOSerializer + + folder_id = models.TextField(blank=True, null=True) + folder_name = models.TextField(blank=True, null=True) + folder_location = models.TextField(blank=True, null=True) + user_settings = models.ForeignKey(UserSettings, null=True, blank=True, on_delete=models.CASCADE) + + @property + def folder_path(self): + return self.folder_name + + @property + def display_name(self): + return u'{0}: {1}'.format(self.config.full_name, self.folder_id) + + def set_folder(self, folder_id, auth): + host = settings.HOST + if not bucket_exists(host, + self.external_account.oauth_key, + self.external_account.oauth_secret, folder_id): + error_message = ('We are having trouble connecting to that bucket. ' + 'Try a different one.') + raise exceptions.InvalidFolderError(error_message) + + self.folder_id = str(folder_id) + self.folder_name = folder_id + self.save() + + self.nodelogger.log(action='bucket_linked', extra={'bucket': str(folder_id)}, save=True) + + def get_folders(self, **kwargs): + # This really gets only buckets, not subfolders, + # as that's all we want to be linkable on a node. + try: + buckets = get_bucket_names(self) + except Exception: + raise exceptions.InvalidAuthError() + + return [ + { + 'addon': SHORT_NAME, + 'kind': 'folder', + 'id': bucket, + 'name': bucket, + 'path': bucket, + 'urls': { + 'folders': '' + } + } + for bucket in buckets + ] + + @property + def complete(self): + return self.has_auth and self.folder_id is not None + + def authorize(self, user_settings, save=False): + self.user_settings = user_settings + self.nodelogger.log(action='node_authorized', save=save) + + def clear_settings(self): + self.folder_id = None + self.folder_name = None + self.folder_location = None + + def deauthorize(self, auth=None, log=True): + """Remove user authorization from this node and log the event.""" + self.clear_settings() + self.clear_auth() # Also performs a save + + if log: + self.nodelogger.log(action='node_deauthorized', save=True) + + def delete(self, save=True): + self.deauthorize(log=False) + super(NodeSettings, self).delete(save=save) + + def serialize_waterbutler_credentials(self): + if not self.has_auth: + raise exceptions.AddonError('Cannot serialize credentials for {} addon'.format(FULL_NAME)) + return { + 'host': settings.HOST, + 'access_key': self.external_account.oauth_key, + 'secret_key': self.external_account.oauth_secret, + } + + def serialize_waterbutler_settings(self): + if not self.folder_id: + raise exceptions.AddonError('Cannot serialize settings for {} addon'.format(FULL_NAME)) + return { + 'bucket': self.folder_id + } + + def create_waterbutler_log(self, auth, action, metadata): + url = self.owner.web_url_for('addon_view_or_download_file', path=metadata['path'], provider=SHORT_NAME) + + self.owner.add_log( + '{0}_{1}'.format(SHORT_NAME, action), + auth=auth, + params={ + 'project': self.owner.parent_id, + 'node': self.owner._id, + 'path': metadata['materialized'], + 'bucket': self.folder_id, + 'urls': { + 'view': url, + 'download': url + '?action=download' + } + }, + ) + + def after_delete(self, user): + self.deauthorize(Auth(user=user), log=True) diff --git a/StorageAddon/osf.io/addon/provider.py b/StorageAddon/osf.io/addon/provider.py new file mode 100644 index 0000000..ae32265 --- /dev/null +++ b/StorageAddon/osf.io/addon/provider.py @@ -0,0 +1,22 @@ +from addons.myminio import SHORT_NAME, FULL_NAME +from addons.myminio.serializer import MyMinIOSerializer + + +class MyMinIOProvider(object): + """An alternative to `ExternalProvider` not tied to OAuth""" + + name = FULL_NAME + short_name = SHORT_NAME + serializer = MyMinIOSerializer + + def __init__(self, account=None): + super(MyMinIOProvider, self).__init__() + + # provide an unauthenticated session by default + self.account = account + + def __repr__(self): + return '<{name}: {status}>'.format( + name=self.__class__.__name__, + status=self.account.provider_id if self.account else 'anonymous' + ) diff --git a/StorageAddon/osf.io/addon/requirements.txt b/StorageAddon/osf.io/addon/requirements.txt new file mode 100644 index 0000000..8e6ed87 --- /dev/null +++ b/StorageAddon/osf.io/addon/requirements.txt @@ -0,0 +1 @@ +boto==2.38.0 diff --git a/StorageAddon/osf.io/addon/routes.py b/StorageAddon/osf.io/addon/routes.py new file mode 100644 index 0000000..ac287eb --- /dev/null +++ b/StorageAddon/osf.io/addon/routes.py @@ -0,0 +1,79 @@ +from addons.myminio import SHORT_NAME +from addons.myminio import views +from framework.routing import Rule, json_renderer + +api_routes = { + 'rules': [ + Rule( + [ + '/settings/{}/accounts/'.format(SHORT_NAME), + ], + 'post', + views.myminio_add_user_account, + json_renderer, + ), + Rule( + [ + '/settings/{}/accounts/'.format(SHORT_NAME), + ], + 'get', + views.myminio_account_list, + json_renderer, + ), + Rule( + [ + '/project//{}/settings/'.format(SHORT_NAME), + '/project//node//{}/settings/'.format(SHORT_NAME), + ], + 'put', + views.myminio_set_config, + json_renderer, + ), + Rule( + [ + '/project//{}/settings/'.format(SHORT_NAME), + '/project//node//{}/settings/'.format(SHORT_NAME), + ], + 'get', + views.myminio_get_config, + json_renderer, + ), + Rule( + [ + '/project//{}/user-auth/'.format(SHORT_NAME), + '/project//node//{}/user-auth/'.format(SHORT_NAME), + ], + 'put', + views.myminio_import_auth, + json_renderer, + ), + Rule( + [ + '/project//{}/user-auth/'.format(SHORT_NAME), + '/project//node//{}/user-auth/'.format(SHORT_NAME), + ], + 'delete', + views.myminio_deauthorize_node, + json_renderer, + ), + Rule( + [ + '/project//{}/buckets/'.format(SHORT_NAME), + '/project//node//{}/buckets/'.format(SHORT_NAME), + ], + 'get', + views.myminio_folder_list, + json_renderer, + ), + Rule( + [ + '/project//{}/newbucket/'.format(SHORT_NAME), + '/project//node//{}/newbucket/'.format(SHORT_NAME), + ], + 'post', + views.myminio_create_bucket, + json_renderer + ), + ], + 'prefix': '/api/v1', +} diff --git a/StorageAddon/osf.io/addon/serializer.py b/StorageAddon/osf.io/addon/serializer.py new file mode 100644 index 0000000..8a50c4a --- /dev/null +++ b/StorageAddon/osf.io/addon/serializer.py @@ -0,0 +1,45 @@ +from addons.base.serializer import StorageAddonSerializer +from addons.myminio import SHORT_NAME +from addons.myminio import settings +from addons.myminio import utils +from website.util import web_url_for + + +class MyMinIOSerializer(StorageAddonSerializer): + addon_short_name = SHORT_NAME + + REQUIRED_URLS = [] + + @property + def addon_serialized_urls(self): + node = self.node_settings.owner + user_settings = self.node_settings.user_settings or self.user_settings + + result = { + 'accounts': node.api_url_for('{}_account_list'.format(SHORT_NAME)), + 'createBucket': node.api_url_for('{}_create_bucket'.format(SHORT_NAME)), + 'importAuth': node.api_url_for('{}_import_auth'.format(SHORT_NAME)), + 'create': node.api_url_for('{}_add_user_account'.format(SHORT_NAME)), + 'deauthorize': node.api_url_for('{}_deauthorize_node'.format(SHORT_NAME)), + 'folders': node.api_url_for('{}_folder_list'.format(SHORT_NAME)), + 'config': node.api_url_for('{}_set_config'.format(SHORT_NAME)), + 'files': node.web_url_for('collect_file_trees'), + } + if user_settings: + result['owner'] = web_url_for('profile_view_id', + uid=user_settings.owner._id) + return result + + def serialized_folder(self, node_settings): + return { + 'path': node_settings.folder_id, + 'name': node_settings.folder_name + } + + def credentials_are_valid(self, user_settings, client=None): + if user_settings: + for account in user_settings.external_accounts.all(): + if utils.can_list(settings.HOST, + account.oauth_key, account.oauth_secret): + return True + return False diff --git a/StorageAddon/osf.io/addon/settings/__init__.py b/StorageAddon/osf.io/addon/settings/__init__.py new file mode 100644 index 0000000..4d3fcfa --- /dev/null +++ b/StorageAddon/osf.io/addon/settings/__init__.py @@ -0,0 +1,8 @@ +import logging +from .defaults import * # noqa + +logger = logging.getLogger(__name__) +try: + from .local import * # noqa +except ImportError: + logger.warn('No local.py settings file found') diff --git a/StorageAddon/osf.io/addon/settings/defaults.py b/StorageAddon/osf.io/addon/settings/defaults.py new file mode 100644 index 0000000..9eb235d --- /dev/null +++ b/StorageAddon/osf.io/addon/settings/defaults.py @@ -0,0 +1 @@ +MAX_FILE_SIZE = 128 # MB diff --git a/StorageAddon/osf.io/addon/settings/local-dist.py b/StorageAddon/osf.io/addon/settings/local-dist.py new file mode 100644 index 0000000..dde3504 --- /dev/null +++ b/StorageAddon/osf.io/addon/settings/local-dist.py @@ -0,0 +1,2 @@ +# MyMinIO app credentials +HOST = 'changeme' diff --git a/StorageAddon/osf.io/addon/static/comicon.png b/StorageAddon/osf.io/addon/static/comicon.png new file mode 100644 index 0000000..7acb4dc Binary files /dev/null and b/StorageAddon/osf.io/addon/static/comicon.png differ diff --git a/StorageAddon/osf.io/addon/static/myminioAnonymousLogActionList.json b/StorageAddon/osf.io/addon/static/myminioAnonymousLogActionList.json new file mode 100644 index 0000000..af3169c --- /dev/null +++ b/StorageAddon/osf.io/addon/static/myminioAnonymousLogActionList.json @@ -0,0 +1,11 @@ +{ + "myminio_bucket_linked" : "A user linked an My MinIO bucket to a project", + "myminio_bucket_unlinked" : "A user unselected an My MinIO bucket in a project", + "myminio_file_added" : "A user added a file to an My MinIO bucket in a project", + "myminio_file_removed" : "A user removed a file in an My MinIO bucket in a project", + "myminio_file_updated" : "A user updated a file in an My MinIO bucket in a project", + "myminio_folder_created" : "A user created a folder in an My MinIO in a project", + "myminio_node_authorized" : "A user authorized the My MinIO addon for a project", + "myminio_node_deauthorized" : "A user deauthorized the My MinIO addon for a project", + "myminio_node_deauthorized_no_user" : "My MinIO addon for a project deauthorized" +} diff --git a/StorageAddon/osf.io/addon/static/myminioLogActionList.json b/StorageAddon/osf.io/addon/static/myminioLogActionList.json new file mode 100644 index 0000000..09de23f --- /dev/null +++ b/StorageAddon/osf.io/addon/static/myminioLogActionList.json @@ -0,0 +1,11 @@ +{ + "myminio_bucket_linked" : "${user} linked the My MinIO bucket ${bucket} to ${node}", + "myminio_bucket_unlinked" : "${user} unselected the My MinIO bucket ${bucket} in ${node}", + "myminio_file_added" : "${user} added file ${path} to My MinIO bucket ${bucket} in ${node}", + "myminio_file_removed" : "${user} removed ${path} in My MinIO bucket ${bucket} in ${node}", + "myminio_file_updated" : "${user} updated file ${path} in My MinIO bucket ${bucket} in ${node}", + "myminio_folder_created" : "${user} created folder ${path} in My MinIO bucket ${bucket} in ${node}", + "myminio_node_authorized" : "${user} authorized the My MinIO addon for ${node}", + "myminio_node_deauthorized" : "${user} deauthorized the My MinIO addon for ${node}", + "myminio_node_deauthorized_no_user" : "My MinIO addon for ${node} deauthorized" +} diff --git a/StorageAddon/osf.io/addon/static/myminioNodeConfig.js b/StorageAddon/osf.io/addon/static/myminioNodeConfig.js new file mode 100644 index 0000000..7c567cd --- /dev/null +++ b/StorageAddon/osf.io/addon/static/myminioNodeConfig.js @@ -0,0 +1,244 @@ +'use strict'; + +var $ = require('jquery'); +var ko = require('knockout'); +var m = require('mithril'); +var bootbox = require('bootbox'); +var Raven = require('raven-js'); +var $osf = require('js/osfHelpers'); +var oop = require('js/oop'); + +var OauthAddonFolderPicker = require('js/oauthAddonNodeConfig')._OauthAddonNodeConfigViewModel; + +var MyMinIOFolderPickerViewModel = oop.extend(OauthAddonFolderPicker, { + constructor: function(addonName, url, selector, folderPicker, opts, tbOpts) { + var self = this; + self.super.constructor(addonName, url, selector, folderPicker, tbOpts); + // Non-OAuth fields + self.accessKey = ko.observable(''); + self.secretKey = ko.observable(''); + // Treebeard config + self.treebeardOptions = $.extend( + {}, + OauthAddonFolderPicker.prototype.treebeardOptions, + { // TreeBeard Options + columnTitles: function() { + return [{ + title: 'Buckets', + width: '75%', + sort: false + }, { + title: 'Select', + width: '25%', + sort: false + }]; + }, + resolveToggle: function(item) { + return ''; + }, + resolveIcon: function(item) { + return m('i.fa.fa-folder-o', ' '); + }, + }, + tbOpts + ); + }, + + connectAccount: function() { + var self = this; + if( !self.accessKey() && !self.secretKey() ){ + self.changeMessage('Please enter both an API access key and secret key.', 'text-danger'); + return; + } + + if (!self.accessKey() ){ + self.changeMessage('Please enter an API access key.', 'text-danger'); + return; + } + + if (!self.secretKey() ){ + self.changeMessage('Please enter an API secret key.', 'text-danger'); + return; + } + $osf.block(); + + return $osf.postJSON( + self.urls().create, { + secret_key: self.secretKey(), + access_key: self.accessKey() + } + ).done(function(response) { + $osf.unblock(); + self.clearModal(); + $('#myminioInputCredentials').modal('hide'); + self.changeMessage('Successfully added My MinIO credentials.', 'text-success', null, true); + self.updateFromData(response); + self.importAuth(); + }).fail(function(xhr, status, error) { + $osf.unblock(); + var message = ''; + var response = JSON.parse(xhr.responseText); + if (response && response.message) { + message = response.message; + } + self.changeMessage(message, 'text-danger'); + Raven.captureMessage('Could not add My MinIO credentials', { + extra: { + url: self.urls().importAuth, + textStatus: status, + error: error + } + }); + }); + }, + /** + * Tests if the given string is a valid My MinIO bucket name. Supports two modes: strict and lax. + * Strict is for bucket creation and follows the guidelines at: + * + * http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules + * + * However, the US East (N. Virginia) region currently permits much laxer naming rules. The S3 + * docs claim this will be changed at some point, but to support our user's already existing + * buckets, we provide the lax mode checking. + * + * Strict checking is the default. + * + * @param {String} bucketName user-provided name of bucket to validate + * @param {Boolean} laxChecking whether to use the more permissive validation + */ + isValidBucketName: function(bucketName, laxChecking) { + if (laxChecking === true) { + return /^[a-zA-Z0-9.\-_]{1,255}$/.test(bucketName); + } + var label = '[a-z0-9]+(?:[a-z0-9\-]*[a-z0-9])?'; + var strictBucketName = new RegExp('^' + label + '(?:\\.' + label + ')*$'); + var isIpAddress = /^[0-9]+(?:\.[0-9]+){3}$/; + return bucketName.length >= 3 && bucketName.length <= 63 && + strictBucketName.test(bucketName) && !isIpAddress.test(bucketName); + }, + + /** Reset all fields from My MinIO credentials input modal */ + clearModal: function() { + var self = this; + self.message(''); + self.messageClass('text-info'); + self.secretKey(null); + self.accessKey(null); + }, + + createBucket: function(self, bucketName) { + $osf.block(); + bucketName = bucketName.toLowerCase(); + return $osf.postJSON( + self.urls().createBucket, {bucket_name: bucketName} + ).done(function(response) { + $osf.unblock(); + self.loadedFolders(false); + self.activatePicker(); + var msg = 'Successfully created bucket "' + $osf.htmlEscape(bucketName) + '". You can now select it from the list.'; + var msgType = 'text-success'; + self.changeMessage(msg, msgType, null, true); + }).fail(function(xhr) { + var resp = JSON.parse(xhr.responseText); + var message = resp.message; + var title = resp.title || 'Problem creating bucket'; + $osf.unblock(); + if (!message) { + message = 'Looks like that name is taken. Try another name?'; + } + bootbox.confirm({ + title: $osf.htmlEscape(title), + message: $osf.htmlEscape(message), + callback: function(result) { + if (result) { + self.openCreateBucket(); + } + }, + buttons:{ + confirm:{ + label:'Try again' + } + } + }); + }); + }, + + openCreateBucket: function() { + var self = this; + + bootbox.dialog({ + title: 'Create a new bucket', + message: + '
' + + '
' + + '
' + + '
' + + ' ' + + '
' + + ' ' + + '
' + + '' + + '
'+ + '
' + + '
' + + '
' + + '
' + + '
', + buttons: { + cancel: { + label: 'Cancel', + className: 'btn-default' + }, + confirm: { + label: 'Create', + className: 'btn-success', + callback: function () { + var bucketName = $('#bucketName').val(); + + if (!bucketName) { + var errorMessage = $('#bucketModalErrorMessage'); + errorMessage.text('Bucket name cannot be empty'); + errorMessage[0].classList.add('text-danger'); + return false; + } else if (!self.isValidBucketName(bucketName, false)) { + bootbox.confirm({ + title: 'Invalid bucket name', + message: 'My MinIO buckets can contain lowercase letters, numbers, and hyphens separated by' + + ' periods. Please try another name.', + callback: function (result) { + if (result) { + self.openCreateBucket(); + } + }, + buttons: { + confirm: { + label: 'Try again' + } + } + }); + } else { + self.createBucket(self, bucketName); + } + } + } + } + }); + } +}); + +// Public API +function MyMinIONodeConfig(addonName, selector, url, folderPicker, opts, tbOpts) { + var self = this; + self.url = url; + self.folderPicker = folderPicker; + opts = opts || {}; + tbOpts = tbOpts || {}; + self.viewModel = new MyMinIOFolderPickerViewModel(addonName, url, selector, folderPicker, opts, tbOpts); + self.viewModel.updateFromData(); + $osf.applyBindings(self.viewModel, selector); +} + +module.exports = { + MyMinIONodeConfig: MyMinIONodeConfig, + _MyNinIONodeConfigViewModel: MyMinIOFolderPickerViewModel +}; diff --git a/StorageAddon/osf.io/addon/static/myminioUserConfig.js b/StorageAddon/osf.io/addon/static/myminioUserConfig.js new file mode 100644 index 0000000..980d65f --- /dev/null +++ b/StorageAddon/osf.io/addon/static/myminioUserConfig.js @@ -0,0 +1,172 @@ +/** +* Module that controls the My MinIO user settings. Includes Knockout view-model +* for syncing data. +*/ + +var ko = require('knockout'); +var $ = require('jquery'); +var Raven = require('raven-js'); +var bootbox = require('bootbox'); +require('js/osfToggleHeight'); + +var language = require('js/osfLanguage').Addons.myminio; +var osfHelpers = require('js/osfHelpers'); +var addonSettings = require('js/addonSettings'); +var ChangeMessageMixin = require('js/changeMessage'); + +var ExternalAccount = addonSettings.ExternalAccount; + +var $modal = $('#myminioInputCredentials'); + + +function ViewModel(url) { + var self = this; + + self.properName = 'My MinIO'; + self.accessKey = ko.observable(); + self.secretKey = ko.observable(); + self.account_url = '/api/v1/settings/myminio/accounts/'; + self.accounts = ko.observableArray(); + + ChangeMessageMixin.call(self); + + /** Reset all fields from My MinIO credentials input modal */ + self.clearModal = function() { + self.message(''); + self.messageClass('text-info'); + self.accessKey(null); + self.secretKey(null); + }; + /** Send POST request to authorize My MinIO */ + self.connectAccount = function() { + // Selection should not be empty + if( !self.accessKey() && !self.secretKey() ){ + self.changeMessage('Please enter both an API access key and secret key.', 'text-danger'); + return; + } + + if (!self.accessKey() ){ + self.changeMessage('Please enter an API access key.', 'text-danger'); + return; + } + + if (!self.secretKey() ){ + self.changeMessage('Please enter an API secret key.', 'text-danger'); + return; + } + return osfHelpers.postJSON( + self.account_url, + ko.toJS({ + access_key: self.accessKey, + secret_key: self.secretKey, + }) + ).done(function() { + self.clearModal(); + $modal.modal('hide'); + self.updateAccounts(); + + }).fail(function(xhr, textStatus, error) { + var errorMessage = (xhr.status === 400 && xhr.responseJSON.message !== undefined) ? xhr.responseJSON.message : language.authError; + self.changeMessage(errorMessage, 'text-danger'); + Raven.captureMessage('Could not authenticate with My MinIO', { + extra: { + url: self.account_url, + textStatus: textStatus, + error: error + } + }); + }); + }; + + self.updateAccounts = function() { + return $.ajax({ + url: url, + type: 'GET', + dataType: 'json' + }).done(function (data) { + self.accounts($.map(data.accounts, function(account) { + var externalAccount = new ExternalAccount(account); + externalAccount.accessKey = account.oauth_key; + externalAccount.secretKey = account.oauth_secret; + return externalAccount; + })); + $('#myminio-header').osfToggleHeight({height: 160}); + }).fail(function(xhr, status, error) { + self.changeMessage(language.userSettingsError, 'text-danger'); + Raven.captureMessage('Error while updating addon account', { + extra: { + url: url, + status: status, + error: error + } + }); + }); + }; + + self.askDisconnect = function(account) { + var self = this; + bootbox.confirm({ + title: 'Disconnect My MinIO Account?', + message: '

' + + 'Are you sure you want to disconnect the My MinIO account ' + + osfHelpers.htmlEscape(account.name) + '? This will revoke access to My MinIO for all projects associated with this account.' + + '

', + callback: function (confirm) { + if (confirm) { + self.disconnectAccount(account); + } + }, + buttons:{ + confirm:{ + label:'Disconnect', + className:'btn-danger' + } + } + }); + }; + + self.disconnectAccount = function(account) { + var self = this; + var url = '/api/v1/oauth/accounts/' + account.id + '/'; + var request = $.ajax({ + url: url, + type: 'DELETE' + }); + request.done(function(data) { + self.updateAccounts(); + }); + request.fail(function(xhr, status, error) { + Raven.captureMessage('Error while removing addon authorization for ' + account.id, { + extra: { + url: url, + status: status, + error: error + } + }); + }); + return request; + }; + + self.selectionChanged = function() { + self.changeMessage('',''); + }; + + self.updateAccounts(); +} + +$.extend(ViewModel.prototype, ChangeMessageMixin.prototype); + +function MyMinIOUserConfig(selector, url) { + // Initialization code + var self = this; + self.selector = selector; + self.url = url; + // On success, instantiate and bind the ViewModel + self.viewModel = new ViewModel(url); + osfHelpers.applyBindings(self.viewModel, self.selector); +} + +module.exports = { + MyMinIOViewModel: ViewModel, + MyMinIOUserConfig: MyMinIOUserConfig +}; diff --git a/StorageAddon/osf.io/addon/static/node-cfg.js b/StorageAddon/osf.io/addon/static/node-cfg.js new file mode 100644 index 0000000..218df4d --- /dev/null +++ b/StorageAddon/osf.io/addon/static/node-cfg.js @@ -0,0 +1,5 @@ +'use strict'; + +var MyMinIONodeConfig = require('./myminioNodeConfig.js').MyMinIONodeConfig; +var url = window.contextVars.node.urls.api + 'myminio/settings/'; +new MyMinIONodeConfig('My MinIO', '#myminioScope', url, '#myminioGrid'); diff --git a/StorageAddon/osf.io/addon/static/user-cfg.js b/StorageAddon/osf.io/addon/static/user-cfg.js new file mode 100644 index 0000000..a21fee0 --- /dev/null +++ b/StorageAddon/osf.io/addon/static/user-cfg.js @@ -0,0 +1,5 @@ +'use strict'; + +var MyMinIOUserConfig = require('./myminioUserConfig.js').MyMinIOUserConfig; +var url = '/api/v1/settings/myminio/accounts/'; +new MyMinIOUserConfig('#myminioAddonScope', url); diff --git a/StorageAddon/osf.io/addon/templates/credentials_modal.mako b/StorageAddon/osf.io/addon/templates/credentials_modal.mako new file mode 100644 index 0000000..a2892a5 --- /dev/null +++ b/StorageAddon/osf.io/addon/templates/credentials_modal.mako @@ -0,0 +1,47 @@ + diff --git a/StorageAddon/osf.io/addon/templates/node_settings.mako b/StorageAddon/osf.io/addon/templates/node_settings.mako new file mode 100644 index 0000000..d10f294 --- /dev/null +++ b/StorageAddon/osf.io/addon/templates/node_settings.mako @@ -0,0 +1,93 @@ +
+ + + <%include file="credentials_modal.mako"/> + +

+ + ${addon_full_name} + + + authorized by + % if not is_registration: + ${_("Disconnect Account")} + % endif + + + + + + ${_("Import Account from Profile")} + + + + + +

+ ${_("Loading ...")} +

+
+ + + + + ${_("Connect Account")} + + +
+

+ +
+
+
+

+ ${_("Current Bucket:")} + + + + + ${_("None")} + +

+ +
+ + +
+ +
+

+ Loading buckets...

+
+
+
+ +
+
+
+
+ ${_("Connect %(folderName)s?") % dict(folderName='“”') | n} +
+
+
+ + +
+
+
+
+
+ +
+ +
+ +
+

+
+
diff --git a/StorageAddon/osf.io/addon/templates/user_settings.mako b/StorageAddon/osf.io/addon/templates/user_settings.mako new file mode 100644 index 0000000..49a00ca --- /dev/null +++ b/StorageAddon/osf.io/addon/templates/user_settings.mako @@ -0,0 +1,46 @@ + +
+ + <%include file="credentials_modal.mako"/> + +

+ + + + ${_("Connect or Reauthorize Account")} + +

+ +
+ + ${_("Disconnect Account")} + +
+ + + + + + + + + + + + + + +
+ + ${_("Private project")} + + + + +
+
+ +
+
diff --git a/StorageAddon/osf.io/addon/tests/__init__.py b/StorageAddon/osf.io/addon/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/StorageAddon/osf.io/addon/tests/conftest.py b/StorageAddon/osf.io/addon/tests/conftest.py new file mode 100644 index 0000000..da9f243 --- /dev/null +++ b/StorageAddon/osf.io/addon/tests/conftest.py @@ -0,0 +1 @@ +from osf_tests.conftest import * # noqa diff --git a/StorageAddon/osf.io/addon/tests/factories.py b/StorageAddon/osf.io/addon/tests/factories.py new file mode 100644 index 0000000..4a90ea0 --- /dev/null +++ b/StorageAddon/osf.io/addon/tests/factories.py @@ -0,0 +1,34 @@ +# -*- coding: utf-8 -*- +"""Factories for the My MinIO addon.""" +import factory +from factory.django import DjangoModelFactory + +from addons.myminio import SHORT_NAME +from addons.myminio.models import ( + UserSettings, + NodeSettings +) +from osf_tests.factories import UserFactory, ProjectFactory, ExternalAccountFactory + + +class MyMinIOAccountFactory(ExternalAccountFactory): + provider = SHORT_NAME + provider_id = factory.Sequence(lambda n: 'id-{0}'.format(n)) + oauth_key = factory.Sequence(lambda n: 'key-{0}'.format(n)) + oauth_secret = factory.Sequence(lambda n: 'secret-{0}'.format(n)) + display_name = 'My MinIO Fake User' + + +class MyMinIOUserSettingsFactory(DjangoModelFactory): + class Meta: + model = UserSettings + + owner = factory.SubFactory(UserFactory) + + +class MyMinIONodeSettingsFactory(DjangoModelFactory): + class Meta: + model = NodeSettings + + owner = factory.SubFactory(ProjectFactory) + user_settings = factory.SubFactory(MyMinIOUserSettingsFactory) diff --git a/StorageAddon/osf.io/addon/tests/test_model.py b/StorageAddon/osf.io/addon/tests/test_model.py new file mode 100644 index 0000000..b32e4f2 --- /dev/null +++ b/StorageAddon/osf.io/addon/tests/test_model.py @@ -0,0 +1,104 @@ +# from nose.tools import * # noqa +import unittest + +import mock +import pytest +from nose.tools import (assert_false, assert_true, + assert_equal, assert_is_none) + +from addons.base.tests.models import ( + OAuthAddonNodeSettingsTestSuiteMixin, + OAuthAddonUserSettingTestSuiteMixin +) +from addons.myminio import SHORT_NAME, FULL_NAME +from addons.myminio import settings +from addons.myminio.models import NodeSettings +from addons.myminio.tests.factories import ( + MyMinIOUserSettingsFactory, + MyMinIONodeSettingsFactory, + MyMinIOAccountFactory +) +from framework.auth import Auth +from osf_tests.factories import ProjectFactory, DraftRegistrationFactory +from tests.base import get_default_metaschema + +pytestmark = pytest.mark.django_db + + +class TestUserSettings(OAuthAddonUserSettingTestSuiteMixin, unittest.TestCase): + + short_name = SHORT_NAME + full_name = FULL_NAME + ExternalAccountFactory = MyMinIOAccountFactory + + +class TestNodeSettings(OAuthAddonNodeSettingsTestSuiteMixin, unittest.TestCase): + + short_name = SHORT_NAME + full_name = FULL_NAME + ExternalAccountFactory = MyMinIOAccountFactory + NodeSettingsFactory = MyMinIONodeSettingsFactory + NodeSettingsClass = NodeSettings + UserSettingsFactory = MyMinIOUserSettingsFactory + + def test_registration_settings(self): + registration = ProjectFactory() + clone, message = self.node_settings.after_register( + self.node, registration, self.user, + ) + assert_is_none(clone) + + def test_before_register_no_settings(self): + self.node_settings.user_settings = None + message = self.node_settings.before_register(self.node, self.user) + assert_false(message) + + def test_before_register_no_auth(self): + self.node_settings.external_account = None + message = self.node_settings.before_register(self.node, self.user) + assert_false(message) + + def test_before_register_settings_and_auth(self): + message = self.node_settings.before_register(self.node, self.user) + assert_true(message) + + @mock.patch('website.archiver.tasks.archive') + def test_does_not_get_copied_to_registrations(self, mock_archive): + registration = self.node.register_node( + schema=get_default_metaschema(), + auth=Auth(user=self.user), + draft_registration=DraftRegistrationFactory(branched_from=self.node), + ) + assert_false(registration.has_addon(SHORT_NAME)) + + ## Overrides ## + + def test_serialize_credentials(self): + self.user_settings.external_accounts[0].provider_id = 'user-11' + self.user_settings.external_accounts[0].oauth_key = 'key-11' + self.user_settings.external_accounts[0].oauth_secret = 'secret-15' + self.user_settings.save() + credentials = self.node_settings.serialize_waterbutler_credentials() + + expected = {'host': settings.HOST, + 'access_key': self.node_settings.external_account.oauth_key, + 'secret_key': self.node_settings.external_account.oauth_secret} + assert_equal(credentials, expected) + + @mock.patch('addons.myminio.models.bucket_exists') + def test_set_folder(self, mock_exists): + mock_exists.return_value = True + folder_id = '1234567890' + self.node_settings.set_folder(folder_id, auth=Auth(self.user)) + self.node_settings.save() + # Bucket was set + assert_equal(self.node_settings.folder_id, folder_id) + assert_equal(self.node_settings.folder_name, folder_id) + # Log was saved + last_log = self.node.logs.latest() + assert_equal(last_log.action, '{0}_bucket_linked'.format(self.short_name)) + + def test_serialize_settings(self): + settings = self.node_settings.serialize_waterbutler_settings() + expected = {'bucket': self.node_settings.folder_id} + assert_equal(settings, expected) diff --git a/StorageAddon/osf.io/addon/tests/test_serializer.py b/StorageAddon/osf.io/addon/tests/test_serializer.py new file mode 100644 index 0000000..4fca7b0 --- /dev/null +++ b/StorageAddon/osf.io/addon/tests/test_serializer.py @@ -0,0 +1,31 @@ +# -*- coding: utf-8 -*- +"""Serializer tests for the My MinIO addon.""" +import mock +import pytest + +from addons.base.tests.serializers import StorageAddonSerializerTestSuiteMixin +from addons.myminio.tests.factories import MyMinIOAccountFactory +from addons.myminio.serializer import MyMinIOSerializer + +from tests.base import OsfTestCase + +pytestmark = pytest.mark.django_db + +class TestMyMinIOSerializer(StorageAddonSerializerTestSuiteMixin, OsfTestCase): + addon_short_name = 'myminio' + Serializer = MyMinIOSerializer + ExternalAccountFactory = MyMinIOAccountFactory + client = None + + def set_provider_id(self, pid): + self.node_settings.folder_id = pid + + def setUp(self): + self.mock_can_list = mock.patch('addons.myminio.serializer.utils.can_list') + self.mock_can_list.return_value = True + self.mock_can_list.start() + super(TestMyMinIOSerializer, self).setUp() + + def tearDown(self): + self.mock_can_list.stop() + super(TestMyMinIOSerializer, self).tearDown() diff --git a/StorageAddon/osf.io/addon/tests/test_view.py b/StorageAddon/osf.io/addon/tests/test_view.py new file mode 100644 index 0000000..275acb2 --- /dev/null +++ b/StorageAddon/osf.io/addon/tests/test_view.py @@ -0,0 +1,278 @@ +# -*- coding: utf-8 -*- +from rest_framework import status as http_status + +from boto.exception import S3ResponseError +import mock +from nose.tools import (assert_equal, assert_equals, + assert_true, assert_in, assert_false) +import pytest + +from framework.auth import Auth +from tests.base import OsfTestCase, get_default_metaschema +from osf_tests.factories import ProjectFactory, AuthUserFactory, DraftRegistrationFactory, InstitutionFactory + +from addons.base.tests.views import ( + OAuthAddonConfigViewsTestCaseMixin +) +from addons.myminio.tests.utils import MyMinIOAddonTestCase +from addons.myminio.utils import validate_bucket_name +import addons.myminio.settings as myminio_settings +from website.util import api_url_for +from admin.rdm_addons.utils import get_rdm_addon_option + +pytestmark = pytest.mark.django_db + +class TestMyMinIOViews(MyMinIOAddonTestCase, OAuthAddonConfigViewsTestCaseMixin, OsfTestCase): + def setUp(self): + self.mock_can_list = mock.patch('addons.myminio.views.utils.can_list') + self.mock_can_list.return_value = True + self.mock_can_list.start() + self.mock_uid = mock.patch('addons.myminio.views.utils.get_user_info') + self.mock_uid.return_value = {'id': '1234567890', 'display_name': 'myminio.user'} + self.mock_uid.start() + self.mock_exists = mock.patch('addons.myminio.views.utils.bucket_exists') + self.mock_exists.return_value = True + self.mock_exists.start() + super(TestMyMinIOViews, self).setUp() + + def tearDown(self): + self.mock_can_list.stop() + self.mock_uid.stop() + self.mock_exists.stop() + super(TestMyMinIOViews, self).tearDown() + + def test_myminio_settings_input_empty_keys(self): + url = self.project.api_url_for('myminio_add_user_account') + rv = self.app.post_json(url, { + 'access_key': '', + 'secret_key': '' + }, auth=self.user.auth, expect_errors=True) + assert_equals(rv.status_int, http_status.HTTP_400_BAD_REQUEST) + assert_in('All the fields above are required.', rv.body.decode()) + + def test_myminio_settings_input_empty_access_key(self): + url = self.project.api_url_for('myminio_add_user_account') + rv = self.app.post_json(url, { + 'access_key': '', + 'secret_key': 'Non-empty-secret-key' + }, auth=self.user.auth, expect_errors=True) + assert_equals(rv.status_int, http_status.HTTP_400_BAD_REQUEST) + assert_in('All the fields above are required.', rv.body.decode()) + + def test_myminio_settings_input_empty_secret_key(self): + url = self.project.api_url_for('myminio_add_user_account') + rv = self.app.post_json(url, { + 'access_key': 'Non-empty-access-key', + 'secret_key': '' + }, auth=self.user.auth, expect_errors=True) + assert_equals(rv.status_int, http_status.HTTP_400_BAD_REQUEST) + assert_in('All the fields above are required.', rv.body.decode()) + + def test_myminio_settings_rdm_addons_denied(self): + institution = InstitutionFactory() + self.user.affiliated_institutions.add(institution) + self.user.save() + rdm_addon_option = get_rdm_addon_option(institution.id, self.ADDON_SHORT_NAME) + rdm_addon_option.is_allowed = False + rdm_addon_option.save() + url = self.project.api_url_for('myminio_add_user_account') + rv = self.app.post_json(url,{ + 'access_key': 'aldkjf', + 'secret_key': 'las' + }, auth=self.user.auth, expect_errors=True) + assert_equal(rv.status_int, http_status.HTTP_403_FORBIDDEN) + assert_in('You are prohibited from using this add-on.', rv.body.decode()) + + def test_myminio_set_bucket_no_settings(self): + user = AuthUserFactory() + self.project.add_contributor(user, save=True) + url = self.project.api_url_for('myminio_set_config') + res = self.app.put_json( + url, {'myminio_bucket': 'hammertofall'}, auth=user.auth, + expect_errors=True + ) + assert_equal(res.status_code, http_status.HTTP_400_BAD_REQUEST) + + def test_myminio_set_bucket_no_auth(self): + + user = AuthUserFactory() + user.add_addon('myminio') + self.project.add_contributor(user, save=True) + url = self.project.api_url_for('myminio_set_config') + res = self.app.put_json( + url, {'myminio_bucket': 'hammertofall'}, auth=user.auth, + expect_errors=True + ) + assert_equal(res.status_code, http_status.HTTP_403_FORBIDDEN) + + def test_myminio_set_bucket_registered(self): + registration = self.project.register_node( + get_default_metaschema(), Auth(self.user), + DraftRegistrationFactory(branched_from=self.project), '' + ) + + url = registration.api_url_for('myminio_set_config') + res = self.app.put_json( + url, {'myminio_bucket': 'hammertofall'}, auth=self.user.auth, + expect_errors=True, + ) + + assert_equal(res.status_code, http_status.HTTP_400_BAD_REQUEST) + + @mock.patch('addons.myminio.views.utils.can_list', return_value=False) + def test_user_settings_cant_list(self, mock_can_list): + url = api_url_for('myminio_add_user_account') + rv = self.app.post_json(url, { + 'access_key': 'aldkjf', + 'secret_key': 'las' + }, auth=self.user.auth, expect_errors=True) + + assert_in('Unable to list buckets.', rv.body.decode()) + assert_equals(rv.status_int, http_status.HTTP_400_BAD_REQUEST) + + def test_myminio_remove_node_settings_owner(self): + url = self.node_settings.owner.api_url_for('myminio_deauthorize_node') + self.app.delete(url, auth=self.user.auth) + result = self.Serializer().serialize_settings(node_settings=self.node_settings, current_user=self.user) + assert_equal(result['nodeHasAuth'], False) + + def test_myminio_remove_node_settings_unauthorized(self): + url = self.node_settings.owner.api_url_for('myminio_deauthorize_node') + ret = self.app.delete(url, auth=None, expect_errors=True) + + assert_equal(ret.status_code, 401) + + def test_myminio_get_node_settings_owner(self): + self.node_settings.set_auth(self.external_account, self.user) + self.node_settings.folder_id = 'bucket' + self.node_settings.save() + url = self.node_settings.owner.api_url_for('myminio_get_config') + res = self.app.get(url, auth=self.user.auth) + + result = res.json['result'] + assert_equal(result['nodeHasAuth'], True) + assert_equal(result['userIsOwner'], True) + assert_equal(result['folder']['path'], self.node_settings.folder_id) + + def test_myminio_get_node_settings_unauthorized(self): + url = self.node_settings.owner.api_url_for('myminio_get_config') + unauthorized = AuthUserFactory() + ret = self.app.get(url, auth=unauthorized.auth, expect_errors=True) + + assert_equal(ret.status_code, 403) + + ## Overrides ## + + @mock.patch('addons.myminio.models.get_bucket_names') + def test_folder_list(self, mock_names): + mock_names.return_value = ['bucket1', 'bucket2'] + super(TestMyMinIOViews, self).test_folder_list() + + @mock.patch('addons.myminio.models.bucket_exists') + def test_set_config(self, mock_exists): + mock_exists.return_value = True + self.node_settings.set_auth(self.external_account, self.user) + url = self.project.api_url_for('{0}_set_config'.format(self.ADDON_SHORT_NAME)) + res = self.app.put_json(url, { + 'selected': self.folder + }, auth=self.user.auth) + assert_equal(res.status_code, http_status.HTTP_200_OK) + self.project.reload() + self.node_settings.reload() + assert_equal( + self.project.logs.latest().action, + '{0}_bucket_linked'.format(self.ADDON_SHORT_NAME) + ) + assert_equal(res.json['result']['folder']['name'], self.node_settings.folder_name) + + +class TestCreateBucket(MyMinIOAddonTestCase, OsfTestCase): + + def setUp(self): + + super(TestCreateBucket, self).setUp() + + self.user = AuthUserFactory() + self.consolidated_auth = Auth(user=self.user) + self.auth = self.user.auth + self.project = ProjectFactory(creator=self.user) + + self.project.add_addon('myminio', auth=self.consolidated_auth) + self.project.creator.add_addon('myminio') + + self.user_settings = self.user.get_addon('myminio') + self.user_settings.access_key = 'We-Will-Rock-You' + self.user_settings.secret_key = 'Idontknowanyqueensongs' + self.user_settings.save() + + self.node_settings = self.project.get_addon('myminio') + self.node_settings.bucket = 'Sheer-Heart-Attack' + self.node_settings.user_settings = self.project.creator.get_addon('myminio') + + self.node_settings.save() + + def test_bad_names(self): + assert_false(validate_bucket_name('')) + assert_false(validate_bucket_name('no')) + assert_false(validate_bucket_name('a' * 64)) + assert_false(validate_bucket_name(' leadingspace')) + assert_false(validate_bucket_name('trailingspace ')) + assert_false(validate_bucket_name('bogus naMe')) + assert_false(validate_bucket_name('.cantstartwithp')) + assert_false(validate_bucket_name('or.endwith.')) + assert_false(validate_bucket_name('..nodoubles')) + assert_false(validate_bucket_name('no_unders_in')) + assert_false(validate_bucket_name('-leadinghyphen')) + assert_false(validate_bucket_name('trailinghyphen-')) + assert_false(validate_bucket_name('Mixedcase')) + assert_false(validate_bucket_name('empty..label')) + assert_false(validate_bucket_name('label-.trailinghyphen')) + assert_false(validate_bucket_name('label.-leadinghyphen')) + assert_false(validate_bucket_name('8.8.8.8')) + assert_false(validate_bucket_name('600.9000.0.28')) + assert_false(validate_bucket_name('no_underscore')) + assert_false(validate_bucket_name('_nounderscoreinfront')) + assert_false(validate_bucket_name('no-underscore-in-back_')) + assert_false(validate_bucket_name('no-underscore-in_the_middle_either')) + + def test_names(self): + assert_true(validate_bucket_name('imagoodname')) + assert_true(validate_bucket_name('still.passing')) + assert_true(validate_bucket_name('can-have-dashes')) + assert_true(validate_bucket_name('kinda.name.spaced')) + assert_true(validate_bucket_name('a-o.valid')) + assert_true(validate_bucket_name('11.12.m')) + assert_true(validate_bucket_name('a--------a')) + assert_true(validate_bucket_name('a' * 63)) + + @mock.patch('addons.myminio.views.utils.create_bucket') + @mock.patch('addons.myminio.views.utils.get_bucket_names') + def test_create_bucket_pass(self, mock_names, mock_make): + mock_make.return_value = True + mock_names.return_value = [ + 'butintheend', + 'it', + 'doesntevenmatter' + ] + url = self.project.api_url_for('myminio_create_bucket') + ret = self.app.post_json( + url, + { + 'bucket_name': 'doesntevenmatter' + }, + auth=self.user.auth + ) + + assert_equal(ret.status_int, http_status.HTTP_200_OK) + assert_equal(ret.json, {}) + + @mock.patch('addons.myminio.views.utils.create_bucket') + def test_create_bucket_fail(self, mock_make): + error = S3ResponseError(418, 'because Im a test') + error.message = 'This should work' + mock_make.side_effect = error + + url = '/api/v1/project/{0}/myminio/newbucket/'.format(self.project._id) + ret = self.app.post_json(url, {'bucket_name': 'doesntevenmatter'}, auth=self.user.auth, expect_errors=True) + + assert_equals(ret.body.decode(), '{"message": "This should work", "title": "Problem connecting to My MinIO"}') diff --git a/StorageAddon/osf.io/addon/tests/utils.py b/StorageAddon/osf.io/addon/tests/utils.py new file mode 100644 index 0000000..e9bbb16 --- /dev/null +++ b/StorageAddon/osf.io/addon/tests/utils.py @@ -0,0 +1,51 @@ +# -*- coding: utf-8 -*- +from nose.tools import (assert_equals, assert_true, assert_false) + +from addons.base.tests.base import OAuthAddonTestCaseMixin, AddonTestCase +from addons.myminio.tests.factories import MyMinIOAccountFactory +from addons.myminio.provider import MyMinIOProvider +from addons.myminio.serializer import MyMinIOSerializer +from addons.myminio import utils + +class MyMinIOAddonTestCase(OAuthAddonTestCaseMixin, AddonTestCase): + + ADDON_SHORT_NAME = 'myminio' + ExternalAccountFactory = MyMinIOAccountFactory + Provider = MyMinIOProvider + Serializer = MyMinIOSerializer + client = None + folder = { + 'path': 'bucket', + 'name': 'bucket', + 'id': 'bucket' + } + + def test_https(self): + connection = utils.connect_myminio(host='securehost', + access_key='a', + secret_key='s') + assert_true(connection.is_secure) + assert_equals(connection.host, 'securehost') + assert_equals(connection.port, 443) + + connection = utils.connect_myminio(host='securehost:443', + access_key='a', + secret_key='s') + assert_true(connection.is_secure) + assert_equals(connection.host, 'securehost') + assert_equals(connection.port, 443) + + def test_http(self): + connection = utils.connect_myminio(host='normalhost:80', + access_key='a', + secret_key='s') + assert_false(connection.is_secure) + assert_equals(connection.host, 'normalhost') + assert_equals(connection.port, 80) + + connection = utils.connect_myminio(host='normalhost:8080', + access_key='a', + secret_key='s') + assert_false(connection.is_secure) + assert_equals(connection.host, 'normalhost') + assert_equals(connection.port, 8080) diff --git a/StorageAddon/osf.io/addon/utils.py b/StorageAddon/osf.io/addon/utils.py new file mode 100644 index 0000000..6dad380 --- /dev/null +++ b/StorageAddon/osf.io/addon/utils.py @@ -0,0 +1,131 @@ +import re + +from boto import exception +from boto.s3.bucket import Bucket +from boto.s3.connection import S3Connection, OrdinaryCallingFormat, NoHostProvided +from rest_framework import status as http_status + +import addons.myminio.settings as settings +from framework.exceptions import HTTPError + + +class MyMinIOConnection(S3Connection): + def __init__(self, aws_access_key_id=None, aws_secret_access_key=None, + is_secure=True, port=None, proxy=None, proxy_port=None, + proxy_user=None, proxy_pass=None, + host=NoHostProvided, debug=0, https_connection_factory=None, + calling_format=None, path='/', + provider='aws', bucket_class=Bucket, security_token=None, + suppress_consec_slashes=True, anon=False, + validate_certs=None, profile_name=None): + super(MyMinIOConnection, self).__init__(aws_access_key_id, + aws_secret_access_key, + is_secure, port, proxy, proxy_port, proxy_user, proxy_pass, + host=host, + debug=debug, https_connection_factory=https_connection_factory, + calling_format=calling_format, + path=path, provider=provider, bucket_class=bucket_class, + security_token=security_token, anon=anon, + validate_certs=validate_certs, profile_name=profile_name) + + def _required_auth_capability(self): + return ['s3'] + + +def connect_myminio(host=None, access_key=None, secret_key=None, node_settings=None): + """Helper to build an MyMinIOConnection object + """ + if host is None: + host = settings.HOST + if node_settings is not None and node_settings.external_account is not None: + access_key = node_settings.external_account.oauth_key + secret_key = node_settings.external_account.oauth_secret + port = 443 + m = re.match(r'^(.+)\:([0-9]+)$', host) + if m is not None: + host = m.group(1) + port = int(m.group(2)) + return MyMinIOConnection(access_key, secret_key, + calling_format=OrdinaryCallingFormat(), + host=host, + port=port, + is_secure=port == 443) + + +def get_bucket_names(node_settings): + try: + buckets = connect_myminio(node_settings=node_settings).get_all_buckets() + except exception.NoAuthHandlerFound: + raise HTTPError(http_status.HTTP_403_FORBIDDEN) + except exception.BotoServerError as e: + raise HTTPError(e.status) + + return [bucket.name for bucket in buckets] + + +def validate_bucket_name(name): + """Make sure the bucket name conforms to Amazon's expectations as described at: + http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules + The laxer rules for US East (N. Virginia) are not supported. + """ + label = r'[a-z0-9]+(?:[a-z0-9\-]*[a-z0-9])?' + validate_name = re.compile('^' + label + '(?:\\.' + label + ')*$') + is_ip_address = re.compile(r'^[0-9]+(?:\.[0-9]+){3}$') + return ( + len(name) >= 3 and len(name) <= 63 and bool(validate_name.match(name)) and not bool(is_ip_address.match(name)) + ) + + +def create_bucket(node_settings, bucket_name): + return connect_myminio(node_settings=node_settings).create_bucket(bucket_name) + + +def bucket_exists(host, access_key, secret_key, bucket_name): + """Tests for the existance of a bucket and if the user + can access it with the given keys + """ + if not bucket_name: + return False + + connection = connect_myminio(host, access_key, secret_key) + + if bucket_name != bucket_name.lower(): + # Must use ordinary calling format for mIxEdCaSe bucket names + # otherwise use the default as it handles bucket outside of the US + connection.calling_format = OrdinaryCallingFormat() + + try: + # Will raise an exception if bucket_name doesn't exist + connect_myminio(host, access_key, secret_key).head_bucket(bucket_name) + except exception.S3ResponseError as e: + if e.status not in (301, 302): + return False + return True + + +def can_list(host, access_key, secret_key): + """Return whether or not a user can list + all buckets accessable by this keys + """ + # Bail out early as boto does not handle getting + # Called with (None, None) + if not (host and access_key and secret_key): + return False + + try: + connect_myminio(host, access_key, secret_key).get_all_buckets() + except exception.S3ResponseError: + return False + return True + + +def get_user_info(host, access_key, secret_key): + """Returns an My MinIO User with .display_name and .id, or None + """ + if not (access_key and secret_key): + return None + + try: + return connect_myminio(host, access_key, secret_key).get_all_buckets().owner + except exception.S3ResponseError: + return None diff --git a/StorageAddon/osf.io/addon/views.py b/StorageAddon/osf.io/addon/views.py new file mode 100644 index 0000000..66afaee --- /dev/null +++ b/StorageAddon/osf.io/addon/views.py @@ -0,0 +1,152 @@ +from boto import exception +from django.core.exceptions import ValidationError +from flask import request +from rest_framework import status as http_status + +import addons.myminio.settings as settings +from addons.base import generic_views +from addons.myminio import SHORT_NAME, FULL_NAME +from addons.myminio import utils +from addons.myminio.serializer import MyMinIOSerializer +from admin.rdm_addons.decorators import must_be_rdm_addons_allowed +from framework.auth.decorators import must_be_logged_in +from framework.exceptions import HTTPError +from osf.models import ExternalAccount +from website.project.decorators import ( + must_have_addon, must_have_permission, + must_be_addon_authorizer, +) + +myminio_account_list = generic_views.account_list( + SHORT_NAME, + MyMinIOSerializer +) + +myminio_import_auth = generic_views.import_auth( + SHORT_NAME, + MyMinIOSerializer +) + +myminio_deauthorize_node = generic_views.deauthorize_node( + SHORT_NAME +) + +myminio_get_config = generic_views.get_config( + SHORT_NAME, + MyMinIOSerializer +) + +def _set_folder(node_addon, folder, auth): + folder_id = folder['id'] + node_addon.set_folder(folder_id, auth=auth) + node_addon.save() + +myminio_set_config = generic_views.set_config( + SHORT_NAME, + FULL_NAME, + MyMinIOSerializer, + _set_folder +) + +@must_have_addon(SHORT_NAME, 'node') +@must_be_addon_authorizer(SHORT_NAME) +def myminio_folder_list(node_addon, **kwargs): + """ Returns all the subsequent folders under the folder id passed. + """ + return node_addon.get_folders() + +@must_be_logged_in +@must_be_rdm_addons_allowed(SHORT_NAME) +def myminio_add_user_account(auth, **kwargs): + """Verifies new external account credentials and adds to user's list""" + host = settings.HOST + try: + access_key = request.json['access_key'] + secret_key = request.json['secret_key'] + except KeyError: + raise HTTPError(http_status.HTTP_400_BAD_REQUEST) + + if not (access_key and secret_key): + return { + 'message': 'All the fields above are required.' + }, http_status.HTTP_400_BAD_REQUEST + + user_info = utils.get_user_info(host, access_key, secret_key) + if not user_info: + return { + 'message': ('Unable to access account.\n' + 'Check to make sure that the above credentials are valid, ' + 'and that they have permission to list buckets.') + }, http_status.HTTP_400_BAD_REQUEST + + if not utils.can_list(host, access_key, secret_key): + return { + 'message': ('Unable to list buckets.\n' + 'Listing buckets is required permission that can be changed via IAM') + }, http_status.HTTP_400_BAD_REQUEST + + try: + account = ExternalAccount( + provider=SHORT_NAME, + provider_name=FULL_NAME, + oauth_key=access_key, + oauth_secret=secret_key, + provider_id=user_info.id, + display_name=user_info.display_name, + ) + account.save() + except ValidationError: + # ... or get the old one + account = ExternalAccount.objects.get( + provider=SHORT_NAME, + provider_id=user_info.id + ) + if account.oauth_key != access_key or account.oauth_secret != secret_key: + account.oauth_key = access_key + account.oauth_secret = secret_key + account.save() + assert account is not None + + if not auth.user.external_accounts.filter(id=account.id).exists(): + auth.user.external_accounts.add(account) + + # Ensure My MinIO is enabled. + auth.user.get_or_add_addon('myminio', auth=auth) + auth.user.save() + + return {} + + +@must_be_addon_authorizer(SHORT_NAME) +@must_have_addon('myminio', 'node') +@must_have_permission('write') +def myminio_create_bucket(auth, node_addon, **kwargs): + bucket_name = request.json.get('bucket_name', '') + # bucket_location = request.json.get('bucket_location', '') + + if not utils.validate_bucket_name(bucket_name): + return { + 'message': 'That bucket name is not valid.', + 'title': 'Invalid bucket name', + }, http_status.HTTP_400_BAD_REQUEST + + try: + # utils.create_bucket(node_addon, bucket_name, bucket_location) + utils.create_bucket(node_addon, bucket_name) + except exception.S3ResponseError as e: + return { + 'message': e.message, + 'title': 'Problem connecting to My MinIO', + }, http_status.HTTP_400_BAD_REQUEST + except exception.S3CreateError as e: + return { + 'message': e.message, + 'title': "Problem creating bucket '{0}'".format(bucket_name), + }, http_status.HTTP_400_BAD_REQUEST + except exception.BotoClientError as e: # Base class catchall + return { + 'message': e.message, + 'title': 'Error connecting to My MinIO', + }, http_status.HTTP_400_BAD_REQUEST + + return {} diff --git a/StorageAddon/osf.io/config/Dockerfile b/StorageAddon/osf.io/config/Dockerfile new file mode 100644 index 0000000..e914981 --- /dev/null +++ b/StorageAddon/osf.io/config/Dockerfile @@ -0,0 +1,197 @@ +FROM node:8-alpine3.9 + +ARG NODE_OPTIONS='--max-old-space-size=4096' + +# Source: https://github.com/docker-library/httpd/blob/7976cabe162268bd5ad2d233d61e340447bfc371/2.4/alpine/Dockerfile#L3 +RUN set -x \ + && addgroup -g 82 -S www-data \ + && adduser -h /var/www -u 82 -D -S -G www-data www-data + +RUN apk add --no-cache --virtual .run-deps \ + su-exec \ + bash \ + python3 \ + git \ + # lxml2 + libxml2 \ + libxslt \ + # psycopg2 + postgresql-libs \ + # cryptography + libffi \ + # gevent + libev \ + libevent \ + openblas-dev \ + && yarn global add bower + +WORKDIR /code + +COPY ./requirements.txt ./ +COPY ./requirements/ ./requirements/ +COPY ./addons/bitbucket/requirements.txt ./addons/bitbucket/ +COPY ./addons/box/requirements.txt ./addons/box/ +#COPY ./addons/citations/requirements.txt ./addons/citations/ +COPY ./addons/dataverse/requirements.txt ./addons/dataverse/ +COPY ./addons/dropbox/requirements.txt ./addons/dropbox/ +#COPY ./addons/dropboxbusiness/requirements.txt ./addons/dropboxbusiness/ +#COPY ./addons/figshare/requirements.txt ./addons/figshare/ +#COPY ./addons/forward/requirements.txt ./addons/forward/ +COPY ./addons/github/requirements.txt ./addons/github/ +COPY ./addons/gitlab/requirements.txt ./addons/gitlab/ +#COPY ./addons/googledrive/requirements.txt ./addons/googledrive/ +#COPY ./addons/iqbrims/requirements.txt ./addons/iqbrims/ +COPY ./addons/mendeley/requirements.txt ./addons/mendeley/ +COPY ./addons/onedrive/requirements.txt /code/addons/onedrive/ +#COPY ./addons/osfstorage/requirements.txt ./addons/osfstorage/ +COPY ./addons/owncloud/requirements.txt ./addons/owncloud/ +COPY ./addons/s3/requirements.txt ./addons/s3/ +COPY ./addons/twofactor/requirements.txt ./addons/twofactor/ +#COPY ./addons/wiki/requirements.txt ./addons/wiki/ +COPY ./addons/zotero/requirements.txt ./addons/zotero/ +COPY ./addons/swift/requirements.txt ./addons/swift/ +COPY ./addons/azureblobstorage/requirements.txt ./addons/azureblobstorage/ +COPY ./addons/weko/requirements.txt ./addons/weko/ +COPY ./addons/s3compat/requirements.txt ./addons/s3compat/ +COPY ./addons/nextcloud/requirements.txt ./addons/nextcloud/ +COPY ./addons/nextcloudinstitutions/requirements.txt ./addons/nextcloudinstitutions/ +COPY ./addons/myminio/requirements.txt ./addons/myminio/ + +RUN set -ex \ + && mkdir -p /var/www \ + && chown www-data:www-data /var/www \ + && apk add --no-cache --virtual .build-deps \ + build-base \ + linux-headers \ + python3-dev \ + # lxml2 + musl-dev \ + libxml2-dev \ + libxslt-dev \ + # psycopg2 + postgresql-dev \ + # cryptography + libffi-dev \ + libpng-dev \ + freetype-dev \ + jpeg-dev \ + && pip3 install Cython \ + && for reqs_file in \ + /code/requirements.txt \ + /code/requirements/release.txt \ + /code/addons/*/requirements.txt \ + ; do \ + pip3 install --no-cache-dir -c /code/requirements/constraints.txt -r "$reqs_file" \ + ; done \ + && (pip3 uninstall uritemplate.py --yes || true) \ + && pip3 install --no-cache-dir uritemplate.py==0.3.0 \ + # Fix: https://github.com/CenterForOpenScience/osf.io/pull/6783 + && python3 -m compileall /usr/lib/python3.6 || true \ + && apk del .build-deps + +# Settings +COPY ./tasks/ ./tasks/ +COPY ./website/settings/ ./website/settings/ +COPY ./api/base/settings/ ./api/base/settings/ +COPY ./website/__init__.py ./website/__init__.py +COPY ./addons.json ./addons.json +RUN mv ./website/settings/local-dist.py ./website/settings/local.py \ + && mv ./api/base/settings/local-dist.py ./api/base/settings/local.py \ + && sed 's/DEBUG_MODE = True/DEBUG_MODE = False/' -i ./website/settings/local.py + +# Bower Assets +COPY ./.bowerrc ./bower.json ./ +COPY ./admin/.bowerrc ./admin/bower.json ./admin/ +RUN \ + # OSF + bower install --production --allow-root \ + && bower cache clean --allow-root \ + # Admin + && cd ./admin \ + && bower install --production --allow-root \ + && bower cache clean --allow-root + +# Webpack Assets +# +## OSF +COPY ./package.json ./.yarnrc ./yarn.lock ./ +COPY ./webpack* ./ +COPY ./website/static/ ./website/static/ +COPY ./scripts/translations/ ./scripts/translations/ +COPY ./website/translations/ ./website/translations/ +COPY ./website/static/js/translations/ ./website/static/js/translations/ +## Admin +COPY ./admin/package.json ./admin/yarn.lock ./admin/ +COPY ./admin/webpack* ./admin/ +COPY ./admin/static/ ./admin/static/ +## Addons +COPY ./addons/bitbucket/static/ ./addons/bitbucket/static/ +COPY ./addons/box/static/ ./addons/box/static/ +COPY ./addons/citations/static/ ./addons/citations/static/ +COPY ./addons/dataverse/static/ ./addons/dataverse/static/ +COPY ./addons/dropbox/static/ ./addons/dropbox/static/ +COPY ./addons/dropboxbusiness/static/ ./addons/dropboxbusiness/static/ +COPY ./addons/figshare/static/ ./addons/figshare/static/ +COPY ./addons/forward/static/ ./addons/forward/static/ +COPY ./addons/github/static/ ./addons/github/static/ +COPY ./addons/gitlab/static/ ./addons/gitlab/static/ +COPY ./addons/googledrive/static/ ./addons/googledrive/static/ +COPY ./addons/mendeley/static/ ./addons/mendeley/static/ +COPY ./addons/onedrive/static/ /code/addons/onedrive/static/ +COPY ./addons/osfstorage/static/ ./addons/osfstorage/static/ +COPY ./addons/owncloud/static/ ./addons/owncloud/static/ +COPY ./addons/s3/static/ ./addons/s3/static/ +COPY ./addons/twofactor/static/ ./addons/twofactor/static/ +COPY ./addons/wiki/static/ ./addons/wiki/static/ +COPY ./addons/zotero/static/ ./addons/zotero/static/ +COPY ./addons/swift/static/ ./addons/swift/static/ +COPY ./addons/azureblobstorage/static/ ./addons/azureblobstorage/static/ +COPY ./addons/weko/static/ ./addons/weko/static/ +COPY ./addons/jupyterhub/static/ ./addons/jupyterhub/static/ +COPY ./addons/s3compat/static/ ./addons/s3compat/static/ +COPY ./addons/nextcloud/static/ ./addons/nextcloud/static/ +COPY ./addons/nextcloudinstitutions/static/ ./addons/nextcloudinstitutions/static/ +COPY ./addons/iqbrims/static/ ./addons/iqbrims/static/ +COPY ./addons/myminio/static/ ./addons/myminio/static/ +RUN \ + # OSF + yarn install --frozen-lockfile \ + && mkdir -p ./website/static/built/ \ + && invoke build_js_config_files \ + && yarn run webpack-prod \ + # Admin + && cd ./admin \ + && yarn install --frozen-lockfile \ + && yarn run webpack-prod \ + && cd ../ \ + # Cleanup + && yarn cache clean \ + && npm cache clean --force + +# Copy the rest of the code over +COPY ./ ./ + +ARG GIT_COMMIT= +ENV GIT_COMMIT ${GIT_COMMIT} + +RUN pybabel compile -d ./website/translations +RUN pybabel compile -D django -d ./admin/translations + +# TODO: Admin/API should fully specify their bower static deps, and not include ./website/static in their defaults.py. +# (this adds an additional 300+mb to the build image) +RUN for module in \ + api.base.settings \ + admin.base.settings \ + ; do \ + export DJANGO_SETTINGS_MODULE=$module \ + && python3 manage.py collectstatic --noinput --no-init-app \ + ; done \ + && for file in \ + ./website/templates/_log_templates.mako \ + ./website/static/built/nodeCategories.json \ + ; do \ + touch $file && chmod o+w $file \ + ; done \ + && rm ./website/settings/local.py ./api/base/settings/local.py + +CMD ["su-exec", "nobody", "invoke", "--list"] diff --git a/StorageAddon/osf.io/config/addons.json b/StorageAddon/osf.io/config/addons.json new file mode 100644 index 0000000..995974c --- /dev/null +++ b/StorageAddon/osf.io/config/addons.json @@ -0,0 +1,146 @@ +{ + "addons": [ + "box", + "dataverse", + "dropbox", + "figshare", + "forward", + "github", + "gitlab", + "mendeley", + "zotero", + "osfstorage", + "owncloud", + "onedrive", + "s3", + "twofactor", + "wiki", + "googledrive", + "bitbucket", + "swift", + "azureblobstorage", + "weko", + "jupyterhub", + "s3compat", + "myminio", + "nextcloud", + "nextcloudinstitutions", + "iqbrims", + "dropboxbusiness" + ], + "addons_default": [ + "osfstorage", + "nextcloudinstitutions", + "dropboxbusiness" + ], + "addons_archivable": { + "osfstorage": "full", + "box": "partial", + "dataverse": "partial", + "dropbox": "partial", + "figshare": "partial", + "forward": "full", + "googledrive": "partial", + "onedrive": "partial", + "github": "partial", + "gitlab": "partial", + "owncloud": "partial", + "s3": "partial", + "wiki": "full", + "bitbucket": "partial", + "swift": "partial", + "azureblobstorage": "partial", + "weko": "partial", + "s3compat": "partial", + "myminio": "partial", + "nextcloud": "partial", + "nextcloudinstitutions": "partial", + "iqbrims": "partial", + "dropboxbusiness": "partial" + }, + "addons_commentable": [ + "osfstorage", + "box", + "dataverse", + "dropbox", + "figshare", + "github", + "gitlab", + "googledrive", + "owncloud", + "s3", + "bitbucket", + "onedrive", + "swift", + "swift", + "azureblobstorage", + "weko", + "s3compat", + "myminio", + "nextcloud", + "nextcloudinstitutions", + "iqbrims", + "dropboxbusiness" + ], + "addons_based_on_ids": [ + "osfstorage", + "box", + "onedrive" + ], + "addons_oauth_no_redirect": [ + "bitbucket" + ], + "addons_description": { + "box": "Box is a file storage add-on. Connect your Box account to a GakuNin RDM project to interact with files hosted on Box via the GakuNin RDM.", + "dataverse": "Dataverse is an open source software application to share, cite, and archive data. Connect your Dataverse account to share your Dataverse datasets via the GakuNin RDM.", + "dropbox": "Dropbox is a file storage add-on. Connect your Dropbox account to a GakuNin RDM project to interact with files hosted on Dropbox via the GakuNin RDM.", + "figshare": "Figshare is an online digital repository. Connect your figshare account to share your figshare files along with other materials in your GakuNin RDM project.", + "forward": "The External Link add-on allows you to provide a link to a website outside of the GakuNin RDM.", + "github": "GitHub is a web-based Git repository hosting service. Connect your GitHub repo to your GakuNin RDM project to share your code alongside other materials in your GakuNin RDM project.", + "gitlab": "GitLab is an open-source web-based Git repository hosting tool and service. Connect your GitLab repo to your GakuNin RDM project to share your code alongside other materials in your GakuNin RDM project.", + "mendeley": "Mendeley is a reference management tool. Connecting Mendeley folders to GakuNin RDM projects allows you and others to view, copy, and download citations that are relevant to your project from the Project Overview page.", + "zotero": "Zotero is a reference management tool. Connecting Zotero folders to GakuNin RDM projects allows you and others to view, copy, and download citations that are relevant to your project from the Project Overview page.", + "osfstorage": "NII Storage is the default storage provider for GakuNin RDM projects.", + "owncloud": "ownCloud is an open source, self-hosted file sync and share app platform. Connect your ownCloud account to a GakuNin RDM project to interact with files hosted on ownCloud via the GakuNin RDM.", + "s3": "Amazon S3 is a file storage add-on. Connect your S3 account to a GakuNin RDM project to interact with files hosted on S3 via the GakuNin RDM.", + "twofactor": "Two-factor authentication is a security add-on. By using two-factor authentication, you'll protect your GakuNin RDM account with both your password and your mobile phone.", + "wiki": "The wiki is a versatile communication tool. Wikis can be used to explain the main points of your project and can contain information like lab notes or contact information.", + "googledrive": "Google Drive is a file storage add-on. Connect your Google Drive account to a GakuNin RDM project to interact with files hosted on Google Drive via the GakuNin RDM.", + "bitbucket": "Bitbucket is a web-based Git repository hosting service. Connect your Bitbucket repo to your GakuNin RDM project to share your code alongside other materials in your GakuNin RDM project.", + "onedrive": "Microsoft OneDrive is a file storage add-on. Connect your Microsoft OneDrive account to a GakuNin RDM project to interact with files hosted on Microsoft OneDrive via the GakuNin RDM.", + "swift": "OpenStack Swift is a file storage add-on. Connect your OpenStack Swift account to a GakuNin RDM project to interact with files hosted on OpenStack Swift via the GakuNin RDM.", + "azureblobstorage": "Azure Blob Storage is a file storage add-on. Connect your Azure Storage account to a GakuNin RDM project to interact with files hosted on Azure Blob Storage (Block Blob) via the GakuNin RDM.", + "weko": "WEKO is an application server to share, archive data.", + "jupyterhub": "Jupyter is a web-based interactive computational environment. Files on a GakuNin RDM project can be imported to/exported from Jupyter", + "s3compat": "S3 Compatible Storage is a file storage add-on. Connect your S3 Compatible Storage account to a GakuNin RDM project to interact with files hosted on S3 Compatible Storage via the GakuNin RDM.", + "myminio": "My MinIO is a file storage add-on. Connect your My MinIO account to a GakuNin RDM project to interact with files hosted on My MinIO via the GakuNin RDM.", + "nextcloud": "Nextcloud is an open source, self-hosted file sync and share app platform. Connect your Nextcloud account to a GakuNin RDM project to interact with files hosted on Nextcloud via the GakuNin RDM.", + "iqbrims": "IQB-RIMS is a file storage add-on. Connect your Google Drive account to a GakuNin RDM project to interact with files hosted on Google Drive via the GakuNin RDM.", + "nextcloudinstitutions": "Nextcloud for Institutions is a file storage add-on for Institutions. Connect your Nextcloud account to a GakuNin RDM project to interact with files hosted on Nextcloud via the GakuNin RDM.", + "dropboxbusiness": "Dropbox Business for Institutions is a file storage add-on for Institutions. Connect your Dropbox Business administrator account to a GakuNin RDM project to interact with files hosted on Dropbox via the GakuNin RDM." + }, + "addons_url": { + "box": "http://www.box.com", + "dataverse": "https://dataverse.harvard.edu/", + "dropbox": "http://www.dropbox.com", + "figshare": "http://www.figshare.com", + "github": "http://www.github.com", + "gitlab": "https://www.gitlab.com", + "mendeley": "http://www.mendeley.com", + "owncloud": "https://owncloud.org/", + "zotero": "http://www.zotero.org", + "s3": "https://aws.amazon.com/s3/", + "googledrive": "https://drive.google.com", + "bitbucket": "https://bitbucket.org/", + "onedrive": "https://onedrive.live.com", + "swift": "https://wiki.openstack.org/wiki/Swift", + "azureblobstorage": "https://azure.microsoft.com/", + "weko": "https://weko.at.nii.ac.jp/", + "jupyterhub": "https://jupyterhub.readthedocs.io/", + "s3compat": "https://aws.amazon.com/s3/", + "myminio": "https://min.io/", + "nextcloud": "https://nextcloud.com/", + "iqbrims": "https://drive.google.com", + "dropboxbusiness": "http://www.dropbox.com" + } +} diff --git a/StorageAddon/osf.io/config/api/base/settings/defaults.py b/StorageAddon/osf.io/config/api/base/settings/defaults.py new file mode 100644 index 0000000..d66c245 --- /dev/null +++ b/StorageAddon/osf.io/config/api/base/settings/defaults.py @@ -0,0 +1,495 @@ +""" +Django settings for api project. + +Generated by 'django-admin startproject' using Django 1.8. + +For more information on this file, see +https://docs.djangoproject.com/en/1.8/topics/settings/ + +For the full list of settings and their values, see +https://docs.djangoproject.com/en/1.8/ref/settings/ +""" + +import os +from future.moves.urllib.parse import urlparse +from website import settings as osf_settings + +BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +# Quick-start development settings - unsuitable for production +# See https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/ + +DATABASES = { + 'default': { + 'CONN_MAX_AGE': 0, + 'ENGINE': 'osf.db.backends.postgresql', # django.db.backends.postgresql + 'NAME': os.environ.get('OSF_DB_NAME', 'osf'), + 'USER': os.environ.get('OSF_DB_USER', 'postgres'), + 'PASSWORD': os.environ.get('OSF_DB_PASSWORD', ''), + 'HOST': os.environ.get('OSF_DB_HOST', '127.0.0.1'), + 'PORT': os.environ.get('OSF_DB_PORT', '5432'), + 'ATOMIC_REQUESTS': True, + 'TEST': { + 'SERIALIZE': False, + }, + }, +} + +DATABASE_ROUTERS = ['osf.db.router.PostgreSQLFailoverRouter', ] +PASSWORD_HASHERS = [ + 'django.contrib.auth.hashers.BCryptSHA256PasswordHasher', + 'django.contrib.auth.hashers.BCryptPasswordHasher', +] + +AUTH_USER_MODEL = 'osf.OSFUser' + +# SECURITY WARNING: keep the secret key used in production secret! +SECRET_KEY = osf_settings.SECRET_KEY + +AUTHENTICATION_BACKENDS = ( + 'api.base.authentication.backends.ODMBackend', + 'guardian.backends.ObjectPermissionBackend', +) + +# SECURITY WARNING: don't run with debug turned on in production! +DEV_MODE = osf_settings.DEV_MODE +DEBUG = osf_settings.DEBUG_MODE +DEBUG_PROPAGATE_EXCEPTIONS = True + +# session: +SESSION_COOKIE_NAME = 'api' +SESSION_COOKIE_SECURE = osf_settings.SECURE_MODE +SESSION_COOKIE_HTTPONLY = osf_settings.SESSION_COOKIE_HTTPONLY +SESSION_COOKIE_SAMESITE = osf_settings.SESSION_COOKIE_SAMESITE + +# csrf: +CSRF_COOKIE_NAME = 'api-csrf' +CSRF_COOKIE_SECURE = osf_settings.SECURE_MODE +CSRF_COOKIE_HTTPONLY = osf_settings.SECURE_MODE + +ALLOWED_HOSTS = [ + '.osf.io', +] + + +# Application definition + +INSTALLED_APPS = ( + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.messages', + 'django.contrib.sessions', + 'django.contrib.staticfiles', + 'django.contrib.admin', + + # 3rd party + 'django_celery_beat', + 'django_celery_results', + 'rest_framework', + 'corsheaders', + 'raven.contrib.django.raven_compat', + 'django_extensions', + 'guardian', + 'storages', + 'waffle', + 'elasticsearch_metrics', + + # OSF + 'osf', + + # Addons + 'addons.osfstorage', + 'addons.bitbucket', + 'addons.box', + 'addons.dataverse', + 'addons.dropbox', + 'addons.figshare', + 'addons.forward', + 'addons.github', + 'addons.gitlab', + 'addons.googledrive', + 'addons.mendeley', + 'addons.onedrive', + 'addons.owncloud', + 'addons.s3', + 'addons.twofactor', + 'addons.wiki', + 'addons.zotero', + 'addons.swift', + 'addons.azureblobstorage', + 'addons.weko', + 'addons.jupyterhub', + 'addons.iqbrims', + 'addons.dropboxbusiness', + 'addons.nextcloudinstitutions', +) + +# local development using https +if osf_settings.SECURE_MODE and DEBUG: + INSTALLED_APPS += ('sslserver',) + +# TODO: Are there more granular ways to configure reporting specifically related to the API? +RAVEN_CONFIG = { + 'tags': {'App': 'api'}, + 'dsn': osf_settings.SENTRY_DSN, + 'release': osf_settings.VERSION, +} + +BULK_SETTINGS = { + 'DEFAULT_BULK_LIMIT': 100, +} + +MAX_PAGE_SIZE = 100 + +REST_FRAMEWORK = { + 'PAGE_SIZE': 10, + 'DEFAULT_RENDERER_CLASSES': ( + 'api.base.renderers.JSONAPIRenderer', + 'api.base.renderers.JSONRendererWithESISupport', + 'api.base.renderers.BrowsableAPIRendererNoForms', + ), + 'DEFAULT_PARSER_CLASSES': ( + 'api.base.parsers.JSONAPIParser', + 'api.base.parsers.JSONAPIParserForRegularJSON', + 'rest_framework.parsers.FormParser', + 'rest_framework.parsers.MultiPartParser', + ), + 'EXCEPTION_HANDLER': 'api.base.exceptions.json_api_exception_handler', + 'DEFAULT_CONTENT_NEGOTIATION_CLASS': 'api.base.content_negotiation.JSONAPIContentNegotiation', + 'DEFAULT_VERSIONING_CLASS': 'api.base.versioning.BaseVersioning', + 'DEFAULT_VERSION': '2.0', + 'ALLOWED_VERSIONS': ( + '2.0', + '2.1', + '2.2', + '2.3', + '2.4', + '2.5', + '2.6', + '2.7', + '2.8', + '2.9', + '2.10', + '2.11', + '2.12', + '2.13', + '2.14', + '2.15', + '2.16', + '2.17', + '2.18', + '2.19', + '2.20', + ), + 'DEFAULT_FILTER_BACKENDS': ('api.base.filters.OSFOrderingFilter',), + 'DEFAULT_PAGINATION_CLASS': 'api.base.pagination.JSONAPIPagination', + 'ORDERING_PARAM': 'sort', + 'DEFAULT_AUTHENTICATION_CLASSES': ( + # Custom auth classes + 'api.base.authentication.drf.OSFBasicAuthentication', + 'api.base.authentication.drf.OSFSessionAuthentication', + 'api.base.authentication.drf.OSFCASAuthentication', + ), + 'DEFAULT_THROTTLE_CLASSES': ( + 'rest_framework.throttling.UserRateThrottle', + 'api.base.throttling.NonCookieAuthThrottle', + 'api.base.throttling.BurstRateThrottle', + ), + 'DEFAULT_THROTTLE_RATES': { + 'user': '10000/day', + 'non-cookie-auth': '100/hour', + 'add-contributor': '10/second', + 'create-guid': '1000/hour', + 'root-anon-throttle': '1000/hour', + 'test-user': '2/hour', + 'test-anon': '1/hour', + 'send-email': '2/minute', + 'burst': '10/second', + }, +} + +# Settings related to CORS Headers addon: allow API to receive authenticated requests from OSF +# CORS plugin only matches based on "netloc" part of URL, so as workaround we add that to the list +CORS_ORIGIN_ALLOW_ALL = False +CORS_ORIGIN_WHITELIST = ( + urlparse(osf_settings.DOMAIN).netloc, + osf_settings.DOMAIN, +) +# This needs to remain True to allow cross origin requests that are in CORS_ORIGIN_WHITELIST to +# use cookies. +CORS_ALLOW_CREDENTIALS = True +# Set dynamically on app init +ORIGINS_WHITELIST = () + +MIDDLEWARE = ( + 'api.base.middleware.DjangoGlobalMiddleware', + 'api.base.middleware.CeleryTaskMiddleware', + 'api.base.middleware.PostcommitTaskMiddleware', + # A profiling middleware. ONLY FOR DEV USE + # Uncomment and add "prof" to url params to recieve a profile for that url + # 'api.base.middleware.ProfileMiddleware', + + # 'django.contrib.sessions.middleware.SessionMiddleware', + 'api.base.middleware.CorsMiddleware', + 'django.middleware.common.CommonMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + # 'django.contrib.auth.middleware.AuthenticationMiddleware', + # 'django.contrib.auth.middleware.SessionAuthenticationMiddleware', + # 'django.contrib.messages.middleware.MessageMiddleware', + 'django.middleware.clickjacking.XFrameOptionsMiddleware', + 'django.middleware.security.SecurityMiddleware', + # 'waffle.middleware.WaffleMiddleware', + 'api.base.middleware.SloanOverrideWaffleMiddleware', # Delete this and uncomment WaffleMiddleware to revert Sloan +) + +TEMPLATES = [ + { + 'BACKEND': 'django.template.backends.django.DjangoTemplates', + 'DIRS': [os.path.join(BASE_DIR, 'templates')], + 'APP_DIRS': True, + }, +] + + +ROOT_URLCONF = 'api.base.urls' +WSGI_APPLICATION = 'api.base.wsgi.application' + + +LANGUAGE_CODE = 'en-us' + +TIME_ZONE = 'UTC' + +USE_I18N = True + +USE_L10N = True + +USE_TZ = True + +# https://django-storages.readthedocs.io/en/latest/backends/gcloud.html +if os.environ.get('GOOGLE_APPLICATION_CREDENTIALS', False): + # Required to interact with Google Cloud Storage + DEFAULT_FILE_STORAGE = 'api.base.storage.RequestlessURLGoogleCloudStorage' + GS_BUCKET_NAME = os.environ.get('GS_BUCKET_NAME', 'cos-osf-stage-cdn-us') + GS_FILE_OVERWRITE = os.environ.get('GS_FILE_OVERWRITE', False) +elif osf_settings.DEV_MODE or osf_settings.DEBUG_MODE: + DEFAULT_FILE_STORAGE = 'api.base.storage.DevFileSystemStorage' + +# https://docs.djangoproject.com/en/1.8/howto/static-files/ + +STATIC_ROOT = os.path.join(BASE_DIR, 'static/vendor') + +API_BASE = 'v2/' +API_PRIVATE_BASE = '_/' +STATIC_URL = '/static/' + +NODE_CATEGORY_MAP = osf_settings.NODE_CATEGORY_MAP + +DEBUG_TRANSACTIONS = DEBUG + +JWT_SECRET = b'osf_api_cas_login_jwt_secret_32b' +JWE_SECRET = b'osf_api_cas_login_jwe_secret_32b' + +ENABLE_VARNISH = osf_settings.ENABLE_VARNISH +ENABLE_ESI = osf_settings.ENABLE_ESI +VARNISH_SERVERS = osf_settings.VARNISH_SERVERS +ESI_MEDIA_TYPES = osf_settings.ESI_MEDIA_TYPES + +ADDONS_FOLDER_CONFIGURABLE = ['box', 'dropbox', 's3', 'googledrive', 'figshare', 'owncloud', 'onedrive', 'swift', 'azureblobstorage', 'weko', 'iqbrims'] +ADDONS_OAUTH = ADDONS_FOLDER_CONFIGURABLE + ['dataverse', 'github', 'bitbucket', 'gitlab', 'mendeley', 'zotero', 'forward'] + +BYPASS_THROTTLE_TOKEN = 'test-token' + +OSF_SHELL_USER_IMPORTS = None + +# Settings for use in the admin +OSF_URL = 'https://osf.io' + +SELECT_FOR_UPDATE_ENABLED = True + +# Disable anonymous user permissions in django-guardian +ANONYMOUS_USER_NAME = None + +# If set to True, automated tests with extra queries will fail. +NPLUSONE_RAISE = False + +# Timestamp - number of requests to send to cloud storages per minute +TS_REQUESTS_PER_MIN = 30 + +# salt used for generating hashids +HASHIDS_SALT = 'pinkhimalayan' + +# django-elasticsearch-metrics +ELASTICSEARCH_DSL = { + 'default': { + 'hosts': os.environ.get('ELASTIC6_URI', '127.0.0.1:9201'), + 'retry_on_timeout': True, + }, +} +# Store yearly indices for time-series metrics +ELASTICSEARCH_METRICS_DATE_FORMAT = '%Y' + +WAFFLE_CACHE_NAME = 'waffle_cache' +STORAGE_USAGE_CACHE_NAME = 'storage_usage' + + +CACHES = { + 'default': { + 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', + }, + STORAGE_USAGE_CACHE_NAME: { + 'BACKEND': 'django.core.cache.backends.db.DatabaseCache', + 'LOCATION': 'osf_cache_table', + }, + WAFFLE_CACHE_NAME: { + 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', + }, +} + +SLOAN_ID_COOKIE_NAME = 'sloan_id' + +EGAP_PROVIDER_NAME = 'EGAP' + +MAX_SIZE_OF_ES_QUERY = 10000 +DEFAULT_ES_NULL_VALUE = 'N/A' + +TRAVIS_ENV = False + +### NII extensions +LOGIN_BY_EPPN = osf_settings.to_bool('LOGIN_BY_EPPN', False) +USER_TIMEZONE = osf_settings.USER_TIMEZONE +USER_LOCALE = osf_settings.USER_LOCALE +CLOUD_GATEWAY_ISMEMBEROF_PREFIX = osf_settings.CLOUD_GATEWAY_ISMEMBEROF_PREFIX +# install-addons.py +INSTALLED_APPS += ('addons.s3compat',) +ADDONS_FOLDER_CONFIGURABLE.append('s3compat') +ADDONS_OAUTH.append('s3compat') +INSTALLED_APPS += ('addons.myminio',) +ADDONS_FOLDER_CONFIGURABLE.append('myminio') +ADDONS_OAUTH.append('myminio') +INSTALLED_APPS += ('addons.nextcloud',) +ADDONS_FOLDER_CONFIGURABLE.append('nextcloud') +ADDONS_OAUTH.append('nextcloud') + +TST_COMMAND_DELIMITER = ' ' +# RSA key generation settings +SSL_GENERATE_KEY = 'openssl' + TST_COMMAND_DELIMITER + \ + 'genrsa' + TST_COMMAND_DELIMITER + \ + '-des3' + TST_COMMAND_DELIMITER + \ + '-out' + TST_COMMAND_DELIMITER + \ + '{0}.key {1}' +SSL_GENERATE_KEY_NOPASS = 'openssl' + TST_COMMAND_DELIMITER + \ + 'rsa' + TST_COMMAND_DELIMITER + \ + '-in' + TST_COMMAND_DELIMITER + \ + '{0}.key' + TST_COMMAND_DELIMITER + \ + '-out' + TST_COMMAND_DELIMITER + \ + '{0}.key.nopass' +SSL_GENERATE_CSR = 'openssl' + TST_COMMAND_DELIMITER + \ + 'req' + TST_COMMAND_DELIMITER + \ + '-new' + TST_COMMAND_DELIMITER + \ + '-key' + TST_COMMAND_DELIMITER + \ + '{0}.key.nopass' + TST_COMMAND_DELIMITER + \ + '-out' + TST_COMMAND_DELIMITER + \ + '{0}.csr' +SSL_GENERATE_SELF_SIGNED = 'openssl' + TST_COMMAND_DELIMITER + \ + 'req' + TST_COMMAND_DELIMITER + \ + '-x509' + TST_COMMAND_DELIMITER + \ + '-nodes' + TST_COMMAND_DELIMITER + \ + '-days' + TST_COMMAND_DELIMITER + \ + '365' + TST_COMMAND_DELIMITER + \ + '-newkey' + TST_COMMAND_DELIMITER + \ + 'rsa:2048' + TST_COMMAND_DELIMITER + \ + '-keyout' + TST_COMMAND_DELIMITER + \ + '{0}.key' + TST_COMMAND_DELIMITER + \ + '-out' + TST_COMMAND_DELIMITER + \ + '{0}.crt' +SSL_PRIVATE_KEY_GENERATION = 'openssl' + TST_COMMAND_DELIMITER + \ + 'genrsa' + TST_COMMAND_DELIMITER + \ + '-out' + TST_COMMAND_DELIMITER + \ + '{0}' + TST_COMMAND_DELIMITER + \ + '{1}' +SSL_PUBLIC_KEY_GENERATION = 'openssl' + TST_COMMAND_DELIMITER + \ + 'rsa' + TST_COMMAND_DELIMITER + \ + '-in' + TST_COMMAND_DELIMITER + \ + '{0}' + TST_COMMAND_DELIMITER + \ + '-pubout' + TST_COMMAND_DELIMITER + \ + '-out' + TST_COMMAND_DELIMITER + \ + '{1}' + +# UserKey Placement destination +KEY_NAME_PRIVATE = 'pvt' +KEY_NAME_PUBLIC = 'pub' +KEY_BIT_VALUE = '3072' +KEY_EXTENSION = '.pem' +KEY_SAVE_PATH = '/user_key_info/' +KEY_NAME_FORMAT = '{0}_{1}_{2}{3}' +PRIVATE_KEY_VALUE = 1 +PUBLIC_KEY_VALUE = 2 +# FreeTSA openation commands +SSL_CREATE_TIMESTAMP_REQUEST = 'openssl' + TST_COMMAND_DELIMITER + \ + 'ts' + TST_COMMAND_DELIMITER + \ + '-query' + TST_COMMAND_DELIMITER + \ + '-data' + TST_COMMAND_DELIMITER + \ + '{0}' + TST_COMMAND_DELIMITER + \ + '-cert' + TST_COMMAND_DELIMITER + \ + '-sha512' +SSL_GET_TIMESTAMP_RESPONSE = 'openssl' + TST_COMMAND_DELIMITER + \ + 'ts' + TST_COMMAND_DELIMITER + \ + '-verify' + TST_COMMAND_DELIMITER + \ + '-data' + TST_COMMAND_DELIMITER + \ + '{0}' + TST_COMMAND_DELIMITER + \ + '-in' + TST_COMMAND_DELIMITER + \ + '{1}' + TST_COMMAND_DELIMITER + \ + '-CAfile' + TST_COMMAND_DELIMITER + \ + '{2}' + +SSL_CREATE_TIMESTAMP_HASH_REQUEST = 'openssl ts -query -digest {digest} {digest_type} -cert' +SSL_GET_TIMESTAMP_HASH_RESPONSE = 'openssl ts -verify {digest_type} -digest {digest} -in {input} -CAfile {ca}' + +# openssl ts verify check value +OPENSSL_VERIFY_RESULT_OK = 'OK' +# timestamp verify rootKey +VERIFY_ROOT_CERTIFICATE = 'root_cert_verifycate.pem' +# timestamp request const +REQUEST_HEADER = {'Content-Type': 'application/timestamp-query'} +TIME_STAMP_AUTHORITY_URL = 'http://eswg.jnsa.org/freetsa' +ERROR_HTTP_STATUS = [400, 401, 402, 403, 500, 502, 503, 504] +REQUEST_TIME_OUT = 5 +RETRY_COUNT = 3 + +# UPKI flag +USE_UPKI = False + +#uPKI operation commands +UPKI_TIMESTAMP_URL = '' +UPKI_CREATE_TIMESTAMP = '' # {0}=target, {1}=out +UPKI_VERIFY_TIMESTAMP = '' # {0}=target, {1}=in +UPKI_CREATE_TIMESTAMP_HASH = '' # {digest_type}, {digest}, {output} +UPKI_VERIFY_TIMESTAMP_HASH = '' # {digest}, {input} +UPKI_VERIFY_INVALID_MSG = 'LPC_ERR_VERIFY_INVALID' + +# TimeStamp Inspection Status +TIME_STAMP_TOKEN_UNCHECKED = 0 +TIME_STAMP_TOKEN_CHECK_SUCCESS = 1 +TIME_STAMP_TOKEN_CHECK_SUCCESS_MSG = 'OK' +TIME_STAMP_TOKEN_CHECK_NG = 2 +TIME_STAMP_TOKEN_CHECK_NG_MSG = 'Fail: file modified.' +TIME_STAMP_TOKEN_CHECK_FILE_NOT_FOUND = 3 +TIME_STAMP_TOKEN_CHECK_FILE_NOT_FOUND_MSG = 'Fail: not inspected.' +TIME_STAMP_TOKEN_NO_DATA = 4 +TIME_STAMP_TOKEN_NO_DATA_MSG = 'Error: some errors has occurred in processing.' +FILE_NOT_EXISTS = 5 +FILE_NOT_EXISTS_MSG = 'Fail: deleted file.' +FILE_NOT_FOUND = 6 +FILE_NOT_FOUND_MSG = 'Fail: file was gone.' +TIME_STAMP_VERIFICATION_ERR = 7 +TIME_STAMP_VERIFICATION_ERR_MSG = 'Error: some errors has occurred in verification.' +TIME_STAMP_STORAGE_DISCONNECTED = 8 +TIME_STAMP_STORAGE_DISCONNECTED_MSG = 'Error: storage disconnected.' +TIME_STAMP_STORAGE_NOT_ACCESSIBLE = 9 +TIME_STAMP_STORAGE_NOT_ACCESSIBLE_MSG = 'Error: storage service connection error occurred.' + +# Quota settings +DEFAULT_MAX_QUOTA = 100 +WARNING_THRESHOLD = 0.9 +BASE_FOR_METRIC_PREFIX = 1000 +SIZE_UNIT_GB = BASE_FOR_METRIC_PREFIX ** 3 +NII_STORAGE_REGION_ID = 1 diff --git a/StorageAddon/osf.io/config/framework/addons/data/addons.json b/StorageAddon/osf.io/config/framework/addons/data/addons.json new file mode 100644 index 0000000..2a80564 --- /dev/null +++ b/StorageAddon/osf.io/config/framework/addons/data/addons.json @@ -0,0 +1,628 @@ +{ + "addons": { + "GitHub": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making a GitHub repo public or private. The GakuNin RDM does not alter the permissions of linked GitHub repos." + }, + "View / download file versions": { + "status": "full", + "text": "GitHub files and their versions can be viewed/downloaded via GakuNin RDM." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in GitHub." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in GitHub." + }, + "Logs": { + "status": "full", + "text": "GitHub dynamically updates GakuNin RDM logs when files are modified outside the GakuNin RDM. Changes to GitHub repos made before the repo is linked to the GakuNin RDM will not be reflected in GakuNin RDM logs." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Github authorization unless the user forking the project is the same user who authorized the Github add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "GitHub content will be registered, but version history will not be copied to the registration." + } + }, + "GitLab": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making a GitLab repo public or private. The GakuNin RDM does not alter the permissions of linked GitLab repos." + }, + "View / download file versions": { + "status": "full", + "text": "GitLab files and their versions can be viewed/downloaded via GakuNin RDM." + }, + "Add / update files": { + "status": "none", + "text": "Adding/updating files in the project via GakuNin RDM is not implemented yet." + }, + "Delete files": { + "status": "none", + "text": "Deleting files via GakuNin RDM is not implemented yet." + }, + "Logs": { + "status": "none", + "text": "GakuNin RDM does not keep track of changes made using Gitlab directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy GitLab authorization unless the user forking the project is the same user who authorized the GitLab add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "GitLab content will be registered, but version history will not be copied to the registration." + } + }, + "Amazon S3": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making an S3 bucket public or private. The GakuNin RDM does not alter the permissions of linked S3 buckets." + }, + "View / download file versions": { + "status": "partial", + "text": "The S3 add-on supports file versions if versioning is enabled for your S3 buckets." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Amazon S3." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Amazon S3." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your S3 buckets through the GakuNin RDM, but not for changes made using S3 directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy S3 authorization unless the user forking the project is the same user who authorized the S3 add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "S3 content will be registered, but version history will not be copied to the registration." + } + }, + "figshare": { + "Permissions": { + "status": "partial", + "text": " Making a GakuNin RDM project public or private is independent of making figshare content public or private. The GakuNin RDM does not alter the permissions of linked figshare content." + }, + "View / download file versions": { + "status": "partial", + "text": "figshare files can be viewed/downloaded via GakuNin RDM, but version history is not supported." + }, + "Add / update files": { + "status": "partial", + "text": "Files can be added but not updated." + }, + "Delete files": { + "status": "partial", + "text": "Private files and filesets can be deleted via GakuNin RDM." + }, + "Logs": { + "status": "partial", + "text": "GakuNin RDM keeps track of changes you make to your figshare content through GakuNin RDM, but not for changes made using figshare directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy figshare authorization unless the user forking the project is the same user who authorized the figshare add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "figshare content will be registered, but version history will not be copied to the registration." + } + }, + "Dropbox": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of Dropbox privacy. The GakuNin RDM does not alter the permissions of linked Dropbox folders." + }, + "View / download file versions": { + "status": "full", + "text": "Dropbox files and their versions can be viewed/downloaded via GakuNin RDM." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Dropbox." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Dropbox." + }, + "Logs": { + "status": "partial", + "text": "GakuNin RDM keeps track of changes you make to your Dropbox content through GakuNin RDM, but not for changes made using Dropbox directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Dropbox authorization unless the user forking the project is the same user who authorized the Dropbox add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "Dropbox content will be registered, but version history will not be copied to the registration." + } + }, + "OneDrive": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of OneDrive privacy. The GakuNin RDM does not alter the permissions of linked OneDrive folders." + }, + "View / download file versions": { + "status": "full", + "text": "OneDrive files and their versions can be viewed/downloaded via GakuNin RDM. OneNote files are unexportable and cannot be downloaded or viewed." + }, + "Add / update files": { + "status": "none", + "text": "The OneDrive add-on is read-only." + }, + "Delete files": { + "status": "none", + "text": "The OneDrive add-on is read-only." + }, + "Logs": { + "status": "partial", + "text": "GakuNin RDM keeps track of changes you make to your OneDrive add-on configuration. It does not track changes to OneDrive content." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy OneDrive authorization unless the user forking the project is the same user who authorized the OneDrive add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "OneDrive content will be registered, but version history will not be copied to the registration. OneNote files are unexportable and will not be archived." + } + }, + "Dataverse": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making a Dataverse study public or private. The GakuNin RDM allows you to release the latest draft version of a Dataverse dataset." + }, + "View / download file versions": { + "status": "partial", + "text": "Files from the latest release of the selected Dataverse study can be viewed/downloaded. GakuNin RDM users with write permissions can view/download draft files as well." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Dataverse." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Dataverse." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your Dataverse studies through the GakuNin RDM, but not for changes made using Dataverse directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Dataverse authorization unless the user forking the project is the same user who authorized the Dataverse add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "Dataverse content will be registered, but version history will not be copied to the registration." + } + }, + "Google Drive": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of Google Drive privacy. The GakuNin RDM does not alter the permissions of linked Google Drive folders." + }, + "View / download file versions": { + "status": "full", + "text": "Google Drive files and their versions can be viewed/downloaded via GakuNin RDM." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Google Drive." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Google Drive." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your Google Drive content through the GakuNin RDM, but not for changes made using Google Drive directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Google Drive authorization unless the user forking the project is the same user who authorized the Google Drive add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "Google Drive content will be registered, but version history will not be copied to the registration." + } + }, + "Box": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of Box privacy. The GakuNin RDM does not alter the permissions of linked Box folders." + }, + "View / download file versions": { + "status": "full", + "text": "Box files and their versions can be viewed/downloaded via GakuNin RDM." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Box." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Box." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your Box content through the GakuNin RDM, but not for changes made using Box directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Box authorization unless the user forking the project is the same user who authorized the Box add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "Box content will be registered, but version history will not be copied to the registration." + } + }, + "Mendeley": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making a Mendeley folder public or private. The GakuNin RDM does not alter the permissions of a linked Mendeley folder." + }, + "View / download file versions": { + "status": "NA" + }, + "Add / update files": { + "status": "NA" + }, + "Delete files": { + "status": "NA" + }, + "Logs": { + "status": "NA" + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Mendeley authorization unless the user forking the project is the same user who authorized the Mendeley add-on in the source project being forked." + }, + "Registering": { + "status": "none", + "text": "Mendeley content will not be registered." + } + }, + "ownCloud": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of ownCloud privacy. The GakuNin RDM does not alter the permissions of linked ownCloud folders." + }, + "View / download file versions": { + "status": "partial", + "text": "ownCloud files can be viewed/downloaded via GakuNin RDM, but version history is not supported by the ownCloud WebDAV API." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in ownCloud." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in ownCloud." + }, + "Logs": { + "status": "partial", + "text": "GakuNin RDM keeps track of changes you make to your ownCloud content through GakuNin RDM, but not for changes made using ownCloud directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy ownCloud authorization unless the user forking the project is the same user who authorized the ownCloud add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "ownCloud content will be registered, but version history will not be copied to the registration." + } + }, + "Zotero": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making a Zotero folder public or private. The GakuNin RDM does not alter the permissions of a linked Zotero folder." + }, + "View / download file versions": { + "status": "NA" + }, + "Add / update files": { + "status": "NA" + }, + "Delete files": { + "status": "NA" + }, + "Logs": { + "status": "NA" + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Zotero authorization unless the user forking the project is the same user who authorized the Zotero add-on in the source project being forked." + }, + "Registering": { + "status": "none", + "text": "Zotero content will not be registered." + } + }, + "Bitbucket": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making a Bitbucket repo public or private. The GakuNin RDM does not alter the permissions of linked Bitbucket repos." + }, + "View / download file versions": { + "status": "full", + "text": "Bitbucket files and their versions can be viewed/downloaded via GakuNin RDM." + }, + "Add / update files": { + "status": "none", + "text": "Bitbucket does not support adding or updating files via their API." + }, + "Delete files": { + "status": "none", + "text": "Bitbucket does not support deleting files via their API." + }, + "Logs": { + "status": "full", + "text": "GakuNin RDM keeps track of changes you make to your Bitbucket content through GakuNin RDM, but not for changes made using Bitbucket directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Bitbucket authorization unless the user forking the project is the same user who authorized the Bitbucket add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "Bitbucket content will be registered, but version history will not be copied to the registration." + } + }, + "OpenStack Swift": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making an Swift container public or private." + }, + "View / download file versions": { + "status": "partial", + "text": "The Swift add-on does not support file versions." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Swift." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Swift." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your Swift containers through the GakuNin RDM, but not for changes made using Swift directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Swift authorization unless the user forking the project is the same user who authorized the Swift add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "Swift object will be registered, but version history will not be copied to the registration." + } + }, + "Azure Blob Storage": { + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making an Azure Blob Storage container public or private." + }, + "View / download file versions": { + "status": "partial", + "text": "The Azure Blob Storage add-on supports only Block Blobs, and does not support file versions." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Azure Blob Storage." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Azure Blob Storage." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your Azure Blob Storage containers through the GakuNin RDM, but not for changes made using Azure Blob Storage directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Azure Blob Storage authorization unless the user forking the project is the same user who authorized the Azure Blob Storage add-on in the source project being forked." + }, + "Registering": { + "status": "none", + "text": "Azure Blob Storage content will not be registered." + } + }, + "WEKO": { + "Permissions": { + "status": "NA" + }, + "View / download file versions": { + "status": "NA" + }, + "Add / update files": { + "status": "partial", + "text": "Adding/updating indices/items in the project via GakuNin RDM will be reflected in WEKO." + }, + "Delete files": { + "status": "partial", + "text": "Items deleted via GakuNin RDM will be deleted in WEKO." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your WEKO index through the GakuNin RDM, but not for changes made using WEKO directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy WEKO authorization unless the user forking the project is the same user who authorized the WEKO add-on in the source project being forked." + }, + "Registering": { + "status": "none", + "text": "WEKO content will not be registered." + } + }, + "S3 Compatible Storage": { + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy S3 Compatible Storage authorization unless the user forking the project is the same user who authorized the S3 Compatible Storage add-on in the source project being forked." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your S3 Compatible Storage buckets through the GakuNin RDM, but not for changes made using S3 Compatible Storage directly." + }, + "Registering": { + "status": "partial", + "text": "S3 Compatible Storage content will be registered, but version history will not be copied to the registration." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in S3 Compatible Storage." + }, + "View / download file versions": { + "status": "partial", + "text": "The S3 add-on supports file versions if versioning is enabled for your S3 Compatible Storage buckets." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in S3 Compatible Storage." + }, + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making an S3 Compatible Storage bucket public or private. The GakuNin RDM does not alter the permissions of linked S3 Compatible Storage buckets." + } + }, + "My MinIO": { + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy My MinIO authorization unless the user forking the project is the same user who authorized the My MinIO add-on in the source project being forked." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your My MinIO buckets through the GakuNin RDM, but not for changes made using My MinIO directly." + }, + "Registering": { + "status": "partial", + "text": "My MinIO content will be registered, but version history will not be copied to the registration." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in My MinIO." + }, + "View / download file versions": { + "status": "partial", + "text": "The S3 add-on supports file versions if versioning is enabled for your My MinIO buckets." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in My MinIO." + }, + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of making an My MinIO bucket public or private. The GakuNin RDM does not alter the permissions of linked My MinIO buckets." + } + }, + "JupyterHub": { + "Permissions": { + "status": "none", + "text": "The GakuNin RDM does not affect the permissions of JupyterHub." + }, + "View / download file versions": { + "status": "none", + "text": "The JupyterHub add-on does not provide Storage Features." + }, + "Add / update files": { + "status": "none", + "text": "The JupyterHub add-on does not provide Storage Features." + }, + "Delete files": { + "status": "none", + "text": "The JupyterHub add-on does not provide Storage Features." + }, + "Logs": { + "status": "none", + "text": "The JupyterHub add-on does not provide Storage Features." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component copies information about linked JupyterHub but the GakuNin RDM does not affect authentication of JupyterHub." + }, + "Registering": { + "status": "none", + "text": "JupyterHub information will not be registered." + } + }, + "Nextcloud": { + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Nextcloud authorization unless the user forking the project is the same user who authorized the Nextcloud add-on in the source project being forked." + }, + "Logs": { + "status": "partial", + "text": "GakuNin RDM keeps track of changes you make to your Nextcloud content through GakuNin RDM, but not for changes made using Nextcloud directly." + }, + "Registering": { + "status": "partial", + "text": "Nextcloud content will be registered, but version history will not be copied to the registration." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Nextcloud." + }, + "View / download file versions": { + "status": "partial", + "text": "Nextcloud files can be viewed/downloaded via GakuNin RDM, but version history is not supported by the Nextcloud WebDAV API." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Nextcloud." + }, + "Permissions": { + "status": "partial", + "text": "Making a GakuNin RDM project public or private is independent of Nextcloud privacy. The GakuNin RDM does not alter the permissions of linked Nextcloud folders." + } + }, + "IQB-RIMS": { + "Permissions": { + "status": "partial", + "text": "Making an GakuNin RDM project public or private is independent of Google Drive privacy. The GakuNin RDM does not alter the permissions of linked Google Drive folders." + }, + "View / download file versions": { + "status": "full", + "text": "IQB-RIMS files and their versions can be viewed/downloaded via GakuNin RDM." + }, + "Add / update files": { + "status": "full", + "text": "Adding/updating files in the project via GakuNin RDM will be reflected in Google Drive." + }, + "Delete files": { + "status": "full", + "text": "Files deleted via GakuNin RDM will be deleted in Google Drive." + }, + "Logs": { + "status": "partial", + "text": "The GakuNin RDM keeps track of changes you make to your Google Drive content through the GakuNin RDM, but not for changes made using Google Drive directly." + }, + "Forking": { + "status": "partial", + "text": "Forking a project or component does not copy Google Drive authorization unless the user forking the project is the same user who authorized the IQB-RIMS add-on in the source project being forked." + }, + "Registering": { + "status": "partial", + "text": "IQB-RIMS content will be registered, but version history will not be copied to the registration." + } + } + }, + "disclaimers": [ + "This add-on connects your GakuNin RDM project to an external service. Use of this service is bound by its terms and conditions. The GakuNin RDM is not responsible for the service or for your use thereof.", + "This add-on allows you to store files using an external service. Files added to this add-on are not stored within the GakuNin RDM." + ] +} diff --git a/StorageAddon/osf.io/config/website/static/storageAddons.json b/StorageAddon/osf.io/config/website/static/storageAddons.json new file mode 100644 index 0000000..02c7156 --- /dev/null +++ b/StorageAddon/osf.io/config/website/static/storageAddons.json @@ -0,0 +1,86 @@ +{ + "bitbucket": { + "fullName": "Bitbucket", + "externalView": true + }, + "box": { + "fullName": "Box", + "externalView": false + }, + "dataverse": { + "fullName": "Dataverse", + "externalView": false + }, + "dropbox": { + "fullName": "Dropbox", + "externalView": false + }, + "figshare": { + "fullName": "figshare", + "externalView": true + }, + "github": { + "fullName": "GitHub", + "externalView": true + }, + "gitlab": { + "fullName": "GitLab", + "externalView": true + }, + "googledrive": { + "fullName": "Google Drive", + "externalView": true + }, + "osfstorage": { + "fullName": "NII Storage", + "externalView": false + }, + "owncloud": { + "fullName": "ownCloud", + "externalView": false + }, + "s3": { + "fullName": "Amazon S3", + "externalView": false + }, + "onedrive": { + "fullName": "OneDrive", + "externalView": true + }, + "swift": { + "fullName": "OpenStack Swift", + "externalView": false + }, + "azureblobstorage": { + "fullName": "Azure Blob Storage", + "externalView": false + }, + "s3compat": { + "fullName": "S3 Compatible Storage", + "externalView": false + }, + "myminio": { + "fullName": "My MinIO", + "externalView": false + }, + "weko": { + "fullName": "WEKO", + "externalView": false + }, + "nextcloud": { + "fullName": "Nextcloud", + "externalView": false + }, + "iqbrims": { + "fullName": "IQB-RIMS", + "externalView": false + }, + "nextcloudinstitutions": { + "fullName": "Nextcloud for Institutions", + "externalView": false + }, + "dropboxbusiness": { + "fullName": "Dropbox Business", + "externalView": false + } +} diff --git a/StorageAddon/waterbutler/config/setup.py b/StorageAddon/waterbutler/config/setup.py new file mode 100644 index 0000000..6d6f02b --- /dev/null +++ b/StorageAddon/waterbutler/config/setup.py @@ -0,0 +1,71 @@ +from setuptools import setup, find_packages + + +def parse_requirements(requirements): + with open(requirements) as f: + return [l.strip('\n') for l in f if l.strip('\n') and not l.startswith('#')] + + +requirements = parse_requirements('requirements.txt') + +# Taken from option 3 of https://packaging.python.org/guides/single-sourcing-package-version/ +version = {} +with open('waterbutler/version.py') as fp: + exec(fp.read(), version) + +setup( + name='waterbutler', + version=version['__version__'], + namespace_packages=['waterbutler', 'waterbutler.auth', 'waterbutler.providers'], + description='WaterButler Storage Server', + author='Center for Open Science', + author_email='contact@cos.io', + url='https://github.com/CenterForOpenScience/waterbutler', + packages=find_packages(exclude=("tests*", )), + package_dir={'waterbutler': 'waterbutler'}, + include_package_data=True, + # install_requires=requirements, + zip_safe=False, + classifiers=[ + 'Natural Language :: English', + 'Intended Audience :: Developers', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: 3.6', + 'Development Status :: 5 - Production/Stable', + 'License :: OSI Approved :: Apache Software License', + ], + provides=[ + 'waterbutler.auth', + 'waterbutler.providers', + ], + entry_points={ + 'waterbutler.auth': [ + 'osf = waterbutler.auth.osf:OsfAuthHandler', + ], + 'waterbutler.providers': [ + 'cloudfiles = waterbutler.providers.cloudfiles:CloudFilesProvider', + 'dropbox = waterbutler.providers.dropbox:DropboxProvider', + 'figshare = waterbutler.providers.figshare:FigshareProvider', + 'filesystem = waterbutler.providers.filesystem:FileSystemProvider', + 'github = waterbutler.providers.github:GitHubProvider', + 'gitlab = waterbutler.providers.gitlab:GitLabProvider', + 'bitbucket = waterbutler.providers.bitbucket:BitbucketProvider', + 'osfstorage = waterbutler.providers.osfstorage:OSFStorageProvider', + 'owncloud = waterbutler.providers.owncloud:OwnCloudProvider', + 's3 = waterbutler.providers.s3:S3Provider', + 'dataverse = waterbutler.providers.dataverse:DataverseProvider', + 'box = waterbutler.providers.box:BoxProvider', + 'googledrive = waterbutler.providers.googledrive:GoogleDriveProvider', + 'onedrive = waterbutler.providers.onedrive:OneDriveProvider', + 'googlecloud = waterbutler.providers.googlecloud:GoogleCloudProvider', + 'swift = waterbutler.providers.swift:SwiftProvider', + 'azureblobstorage = waterbutler.providers.azureblobstorage:AzureBlobStorageProvider', + 'weko = waterbutler.providers.weko:WEKOProvider', + 's3compat = waterbutler.providers.s3compat:S3CompatProvider', + 'myminio = waterbutler.providers.myminio:MyMinIOProvider', + 'nextcloud = waterbutler.providers.nextcloud:NextcloudProvider', + 'iqbrims = waterbutler.providers.iqbrims:IQBRIMSProvider', + 'dropboxbusiness = waterbutler.providers.dropboxbusiness:DropboxBusinessProvider', + ], + }, +) diff --git a/StorageAddon/waterbutler/provider/__init__.py b/StorageAddon/waterbutler/provider/__init__.py new file mode 100644 index 0000000..4265d52 --- /dev/null +++ b/StorageAddon/waterbutler/provider/__init__.py @@ -0,0 +1 @@ +from .provider import MyMinIOProvider # noqa diff --git a/StorageAddon/waterbutler/provider/metadata.py b/StorageAddon/waterbutler/provider/metadata.py new file mode 100644 index 0000000..d1d8025 --- /dev/null +++ b/StorageAddon/waterbutler/provider/metadata.py @@ -0,0 +1,132 @@ +import os + +from waterbutler.core import metadata + + +class MyMinIOMetadata(metadata.BaseMetadata): + + @property + def provider(self): + return 'myminio' + + @property + def name(self): + return os.path.split(self.path)[1] + + +class MyMinIOFileMetadataHeaders(MyMinIOMetadata, metadata.BaseFileMetadata): + + def __init__(self, path, headers): + self._path = path + # Cast to dict to clone as the headers will + # be destroyed when the request leaves scope + super().__init__(dict(headers)) + + @property + def path(self): + return '/' + self._path + + @property + def size(self): + return self.raw['Content-Length'] + + @property + def content_type(self): + return self.raw['Content-Type'] + + @property + def modified(self): + return self.raw['Last-Modified'] + + @property + def created_utc(self): + return None + + @property + def etag(self): + return self.raw['Etag'].replace('"', '') + + @property + def extra(self): + return { + 'md5': self.raw['Etag'].replace('"', ''), + 'encryption': self.raw.get('X-Amz-Request-Id', '') + } + + +class MyMinIOFileMetadata(MyMinIOMetadata, metadata.BaseFileMetadata): + + @property + def path(self): + return '/' + self.raw['Key'] + + @property + def size(self): + return int(self.raw['Size']) + + @property + def modified(self): + return self.raw['LastModified'] + + @property + def created_utc(self): + return None + + @property + def content_type(self): + return None + + @property + def etag(self): + return self.raw['ETag'].replace('"', '') + + @property + def extra(self): + return { + 'md5': self.raw['ETag'].replace('"', '') + } + + +class MyMinIOFolderKeyMetadata(MyMinIOMetadata, metadata.BaseFolderMetadata): + + @property + def name(self): + return self.raw['Key'].split('/')[-2] + + @property + def path(self): + return '/' + self.raw['Key'] + + +class MyMinIOFolderMetadata(MyMinIOMetadata, metadata.BaseFolderMetadata): + + @property + def name(self): + return self.raw['Prefix'].split('/')[-2] + + @property + def path(self): + return '/' + self.raw['Prefix'] + + +class MyMinIORevision(metadata.BaseFileRevisionMetadata): + + @property + def version_identifier(self): + return 'version' + + @property + def version(self): + if self.raw['IsLatest'] == 'true': + return 'Latest' + return self.raw['VersionId'] + + @property + def modified(self): + return self.raw['LastModified'] + + @property + def extra(self): + return { + 'md5': self.raw['ETag'].replace('"', '') + } diff --git a/StorageAddon/waterbutler/provider/provider.py b/StorageAddon/waterbutler/provider/provider.py new file mode 100644 index 0000000..b4937df --- /dev/null +++ b/StorageAddon/waterbutler/provider/provider.py @@ -0,0 +1,450 @@ +import os +import hashlib +import functools +from urllib import parse +import re + +import xmltodict + +from boto.s3.connection import S3Connection, OrdinaryCallingFormat, NoHostProvided +from boto.connection import HTTPRequest +from boto.s3.bucket import Bucket + +from waterbutler.core import streams +from waterbutler.core import provider +from waterbutler.core import exceptions +from waterbutler.core.path import WaterButlerPath + +from waterbutler.providers.myminio import settings +from waterbutler.providers.myminio.metadata import MyMinIORevision +from waterbutler.providers.myminio.metadata import MyMinIOFileMetadata +from waterbutler.providers.myminio.metadata import MyMinIOFolderMetadata +from waterbutler.providers.myminio.metadata import MyMinIOFolderKeyMetadata +from waterbutler.providers.myminio.metadata import MyMinIOFileMetadataHeaders + + +class MyMinIOConnection(S3Connection): + def __init__(self, aws_access_key_id=None, aws_secret_access_key=None, + is_secure=True, port=None, proxy=None, proxy_port=None, + proxy_user=None, proxy_pass=None, + host=NoHostProvided, debug=0, https_connection_factory=None, + calling_format=None, path='/', + provider='aws', bucket_class=Bucket, security_token=None, + suppress_consec_slashes=True, anon=False, + validate_certs=None, profile_name=None): + super(MyMinIOConnection, self).__init__(aws_access_key_id, + aws_secret_access_key, + is_secure, port, proxy, proxy_port, proxy_user, proxy_pass, + host=host, + debug=debug, https_connection_factory=https_connection_factory, + calling_format=calling_format, + path=path, provider=provider, bucket_class=bucket_class, + security_token=security_token, anon=anon, + validate_certs=validate_certs, profile_name=profile_name) + + def add_auth(self, method, url, headers): + urlo = parse.urlparse(url) + self._auth_handler.add_auth(HTTPRequest(method, urlo.scheme, + urlo.hostname, self.port, + urlo.path, urlo.path, {}, + headers, '')) + return url[:url.index('?')] if '?' in url else url + + def _required_auth_capability(self): + return ['s3'] + + +class MyMinIOProvider(provider.BaseProvider): + """Provider for My MinIO service. + + API docs: http://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html + + Quirks: + + * On S3, folders are not first-class objects, but are instead inferred + from the names of their children. A regular DELETE request issued + against a folder will not work unless that folder is completely empty. + To fully delete an occupied folder, we must delete all of the comprising + objects. Amazon provides a bulk delete operation to simplify this. + + * A GET prefix query against a non-existent path returns 200 + """ + NAME = 'myminio' + + def __init__(self, auth, credentials, settings): + """ + .. note:: + + Neither `MyMinIOConnection#__init__` nor `MyMinIOConnection#get_bucket` + sends a request. + + :param dict auth: Not used + :param dict credentials: Dict containing `access_key` and `secret_key` + :param dict settings: Dict containing `bucket` + """ + super().__init__(auth, credentials, settings) + + host = credentials['host'] + port = 443 + m = re.match(r'^(.+)\:([0-9]+)$', host) + if m is not None: + host = m.group(1) + port = int(m.group(2)) + self.connection = MyMinIOConnection(credentials['access_key'], + credentials['secret_key'], + calling_format=OrdinaryCallingFormat(), + host=host, + port=port, + is_secure=port == 443) + self.bucket = self.connection.get_bucket(settings['bucket'], validate=False) + + async def validate_v1_path(self, path, **kwargs): + if path == '/': + return WaterButlerPath(path) + + implicit_folder = path.endswith('/') + + if implicit_folder: + params = {'prefix': path, 'delimiter': '/'} + resp = await self.make_request( + 'GET', + functools.partial(self.bucket.generate_url, settings.TEMP_URL_SECS, 'GET', query_parameters=params), + params=params, + expects=(200, 404), + throws=exceptions.MetadataError, + ) + else: + resp = await self.make_request( + 'HEAD', + functools.partial(self.bucket.new_key(path).generate_url, settings.TEMP_URL_SECS, 'HEAD'), + expects=(200, 404), + throws=exceptions.MetadataError, + ) + + await resp.release() + + if resp.status == 404: + raise exceptions.NotFoundError(str(path)) + + return WaterButlerPath(path) + + async def validate_path(self, path, **kwargs): + return WaterButlerPath(path) + + def can_duplicate_names(self): + return True + + def can_intra_copy(self, dest_provider, path=None): + return type(self) == type(dest_provider) and not getattr(path, 'is_dir', False) + + def can_intra_move(self, dest_provider, path=None): + return type(self) == type(dest_provider) and not getattr(path, 'is_dir', False) + + async def intra_copy(self, dest_provider, source_path, dest_path): + """Copy key from one My MinIO bucket to another. The credentials specified in + `dest_provider` must have read access to `source.bucket`. + """ + exists = await dest_provider.exists(dest_path) + + dest_key = dest_provider.bucket.new_key(dest_path.path) + + # ensure no left slash when joining paths + source_path = '/' + os.path.join(self.settings['bucket'], source_path.path) + headers = {'x-amz-copy-source': parse.quote(source_path)} + url = functools.partial( + dest_key.generate_url, + settings.TEMP_URL_SECS, + 'PUT', + headers=headers, + ) + resp = await self.make_request( + 'PUT', url, + skip_auto_headers={'CONTENT-TYPE'}, + headers=headers, + expects=(200, ), + throws=exceptions.IntraCopyError, + ) + await resp.release() + return (await dest_provider.metadata(dest_path)), not exists + + async def download(self, path, accept_url=False, version=None, range=None, **kwargs): + """Returns a ResponseWrapper (Stream) for the specified path + raises FileNotFoundError if the status from S3 is not 200 + + :param str path: Path to the key you want to download + :param dict \*\*kwargs: Additional arguments that are ignored + :rtype: :class:`waterbutler.core.streams.ResponseStreamReader` + :raises: :class:`waterbutler.core.exceptions.DownloadError` + """ + if not path.is_file: + raise exceptions.DownloadError('No file specified for download', code=400) + + if not version or version.lower() == 'latest': + query_parameters = None + else: + query_parameters = {'versionId': version} + + if kwargs.get('displayName'): + response_headers = {'response-content-disposition': 'attachment; filename*=UTF-8\'\'{}'.format(parse.quote(kwargs['displayName']))} + else: + response_headers = {'response-content-disposition': 'attachment'} + + url = functools.partial( + self.bucket.new_key(path.path).generate_url, + settings.TEMP_URL_SECS, + query_parameters=query_parameters, + response_headers=response_headers + ) + + headers = {} + raw_url = self.connection.add_auth('GET', url('GET'), headers) + + resp = await self.make_request( + 'GET', + raw_url, + range=range, + headers=headers, + expects=(200, 206), + throws=exceptions.DownloadError, + ) + + return streams.ResponseStreamReader(resp) + + async def upload(self, stream, path, conflict='replace', **kwargs): + """Uploads the given stream to My MinIO + + :param waterbutler.core.streams.RequestWrapper stream: The stream to put to My MinIO + :param str path: The full path of the key to upload to/into + + :rtype: dict, bool + """ + path, exists = await self.handle_name_conflict(path, conflict=conflict) + stream.add_writer('md5', streams.HashStreamWriter(hashlib.md5)) + + headers = {'Content-Length': str(stream.size)} + + upload_url = functools.partial( + self.bucket.new_key(path.path).generate_url, + settings.TEMP_URL_SECS, + 'PUT', + headers=headers, + ) + resp = await self.make_request( + 'PUT', + upload_url, + data=stream, + skip_auto_headers={'CONTENT-TYPE'}, + headers=headers, + expects=(200, 201, ), + throws=exceptions.UploadError, + ) + + await resp.release() + return (await self.metadata(path, **kwargs)), not exists + + async def delete(self, path, confirm_delete=0, **kwargs): + """Deletes the key at the specified path + + :param str path: The path of the key to delete + :param int confirm_delete: Must be 1 to confirm root folder delete + """ + if path.is_root: + if not confirm_delete == 1: + raise exceptions.DeleteError( + 'confirm_delete=1 is required for deleting root provider folder', + code=400 + ) + + if path.is_file: + resp = await self.make_request( + 'DELETE', + self.bucket.new_key(path.path).generate_url(settings.TEMP_URL_SECS, 'DELETE'), + expects=(200, 204, ), + throws=exceptions.DeleteError, + ) + await resp.release() + else: + await self._delete_folder(path, **kwargs) + + async def _delete_folder(self, path, **kwargs): + """Query for recursive contents of folder and delete in batches of 1000 + + Called from: func: delete if not path.is_file + + Calls: func: self.make_request + func: self.bucket.generate_url + + :param *ProviderPath path: Path to be deleted + + On S3, folders are not first-class objects, but are instead inferred + from the names of their children. A regular DELETE request issued + against a folder will not work unless that folder is completely empty. + To fully delete an occupied folder, we must delete all of the comprising + objects. Amazon provides a bulk delete operation to simplify this. + """ + more_to_come = True + content_keys = [] + query_params = {'prefix': path.path} + marker = None + + while more_to_come: + if marker is not None: + query_params['marker'] = marker + + resp = await self.make_request( + 'GET', + self.bucket.generate_url(settings.TEMP_URL_SECS, 'GET', query_parameters=query_params), + params=query_params, + expects=(200, ), + throws=exceptions.MetadataError, + ) + + contents = await resp.read() + parsed = xmltodict.parse(contents, strip_whitespace=False)['ListBucketResult'] + more_to_come = parsed.get('IsTruncated') == 'true' + contents = parsed.get('Contents', []) + + if isinstance(contents, dict): + contents = [contents] + + content_keys.extend([content['Key'] for content in contents]) + if len(content_keys) > 0: + marker = content_keys[-1] + + # Query against non-existant folder does not return 404 + if len(content_keys) == 0: + raise exceptions.NotFoundError(str(path)) + + for content_key in content_keys[::-1]: + resp = await self.make_request( + 'DELETE', + self.bucket.new_key(content_key).generate_url(settings.TEMP_URL_SECS, 'DELETE'), + expects=(200, 204, ), + throws=exceptions.DeleteError, + ) + await resp.release() + + async def revisions(self, path, **kwargs): + """Get past versions of the requested key + + :param str path: The path to a key + :rtype list: + """ + query_params = {'prefix': path.path, 'delimiter': '/', 'versions': ''} + url = functools.partial(self.bucket.generate_url, settings.TEMP_URL_SECS, 'GET', query_parameters=query_params) + resp = await self.make_request( + 'GET', + url, + params=query_params, + expects=(200, ), + throws=exceptions.MetadataError, + ) + content = await resp.read() + versions = xmltodict.parse(content)['ListVersionsResult'].get('Version') or [] + + if isinstance(versions, dict): + versions = [versions] + + return [ + MyMinIORevision(item) + for item in versions + if item['Key'] == path.path + ] + + async def metadata(self, path, revision=None, **kwargs): + """Get Metadata about the requested file or folder + + :param WaterButlerPath path: The path to a key or folder + :rtype: dict or list + """ + if path.is_dir: + return (await self._metadata_folder(path)) + + return (await self._metadata_file(path, revision=revision)) + + async def create_folder(self, path, folder_precheck=True, **kwargs): + """ + :param str path: The path to create a folder at + """ + WaterButlerPath.validate_folder(path) + + if folder_precheck: + if (await self.exists(path)): + raise exceptions.FolderNamingConflict(path.name) + + async with self.request( + 'PUT', + functools.partial(self.bucket.new_key(path.path).generate_url, settings.TEMP_URL_SECS, 'PUT'), + skip_auto_headers={'CONTENT-TYPE'}, + expects=(200, 201), + throws=exceptions.CreateFolderError + ): + return MyMinIOFolderMetadata({'Prefix': path.path}) + + async def _metadata_file(self, path, revision=None): + if revision == 'Latest': + revision = None + resp = await self.make_request( + 'HEAD', + functools.partial( + self.bucket.new_key(path.path).generate_url, + settings.TEMP_URL_SECS, + 'HEAD', + query_parameters={'versionId': revision} if revision else None + ), + expects=(200, ), + throws=exceptions.MetadataError, + ) + await resp.release() + return MyMinIOFileMetadataHeaders(path.path, resp.headers) + + async def _metadata_folder(self, path): + params = {'prefix': path.path, 'delimiter': '/'} + resp = await self.make_request( + 'GET', + functools.partial(self.bucket.generate_url, settings.TEMP_URL_SECS, 'GET', query_parameters=params), + params=params, + expects=(200, ), + throws=exceptions.MetadataError, + ) + + contents = await resp.read() + + parsed = xmltodict.parse(contents, strip_whitespace=False)['ListBucketResult'] + + contents = parsed.get('Contents', []) + prefixes = parsed.get('CommonPrefixes', []) + + if not contents and not prefixes and not path.is_root: + # If contents and prefixes are empty then this "folder" + # must exist as a key with a / at the end of the name + # if the path is root there is no need to test if it exists + resp = await self.make_request( + 'HEAD', + functools.partial(self.bucket.new_key(path.path).generate_url, settings.TEMP_URL_SECS, 'HEAD'), + expects=(200, ), + throws=exceptions.MetadataError, + ) + await resp.release() + + if isinstance(contents, dict): + contents = [contents] + + if isinstance(prefixes, dict): + prefixes = [prefixes] + + items = [ + MyMinIOFolderMetadata(item) + for item in prefixes + ] + + for content in contents: + if content['Key'] == path.path: + continue + + if content['Key'].endswith('/'): + items.append(MyMinIOFolderKeyMetadata(content)) + else: + items.append(MyMinIOFileMetadata(content)) + + return items diff --git a/StorageAddon/waterbutler/provider/settings.py b/StorageAddon/waterbutler/provider/settings.py new file mode 100644 index 0000000..5a1b22f --- /dev/null +++ b/StorageAddon/waterbutler/provider/settings.py @@ -0,0 +1,6 @@ +from waterbutler import settings + +config = settings.child('MYMINIO_PROVIDER_CONFIG') + + +TEMP_URL_SECS = int(config.get('TEMP_URL_SECS', 100)) diff --git a/StorageAddon/waterbutler/tests/__init__.py b/StorageAddon/waterbutler/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/StorageAddon/waterbutler/tests/test_provider.py b/StorageAddon/waterbutler/tests/test_provider.py new file mode 100644 index 0000000..62ea40f --- /dev/null +++ b/StorageAddon/waterbutler/tests/test_provider.py @@ -0,0 +1,758 @@ +import pytest + +import io +import time +import base64 +import hashlib +from http import client +from unittest import mock + +import aiohttpretty + +from waterbutler.core import streams +from waterbutler.core import metadata +from waterbutler.core import exceptions +from waterbutler.core.path import WaterButlerPath + +from waterbutler.providers.myminio import MyMinIOProvider +from waterbutler.providers.myminio.metadata import MyMinIOFileMetadata +from waterbutler.providers.myminio.metadata import MyMinIOFolderMetadata + + +@pytest.fixture +def auth(): + return { + 'name': 'cat', + 'email': 'cat@cat.com', + } + + +@pytest.fixture +def credentials(): + return { + 'host': 'Target Host', + 'access_key': 'Dont dead', + 'secret_key': 'open inside', + } + + +@pytest.fixture +def settings(): + return { + 'bucket': 'that kerning' + } + +@pytest.fixture +def mock_time(monkeypatch): + mock_time = mock.Mock(return_value=1454684930.0) + monkeypatch.setattr(time, 'time', mock_time) + + +@pytest.fixture +def provider(auth, credentials, settings): + return MyMinIOProvider(auth, credentials, settings) + + +@pytest.fixture +def file_content(): + return b'sleepy' + + +@pytest.fixture +def file_like(file_content): + return io.BytesIO(file_content) + + +@pytest.fixture +def file_stream(file_like): + return streams.FileStreamReader(file_like) + + +@pytest.fixture +def folder_metadata(): + return b''' + + bucket + + + 1000 + false + + my-image.jpg + 2009-10-12T17:50:30.000Z + "fba9dede5f27731c9771645a39863328" + 434234 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + + my-third-image.jpg + 2009-10-12T17:50:30.000Z + "1b2cf535f27731c974343645a3985328" + 64994 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + + photos/ + + ''' + + +@pytest.fixture +def just_a_folder_metadata(): + return b''' + + bucket + + + 1000 + false + + naptime/ + 2009-10-12T17:50:30.000Z + "fba9dede5f27731c9771645a39863328" + 0 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + ''' + + +@pytest.fixture +def contents_and_self(): + return b''' + + bucket + + + 1000 + false + + thisfolder/ + 2009-10-12T17:50:30.000Z + "fba9dede5f27731c9771645a39863328" + 0 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + + thisfolder/item1 + 2009-10-12T17:50:30.000Z + "fba9dede5f27731c9771645a39863328" + 0 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + + thisfolder/item2 + 2009-10-12T17:50:30.000Z + "fba9dede5f27731c9771645a39863328" + 0 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + ''' + + +@pytest.fixture +def folder_empty_metadata(): + return b''' + + bucket + + + 1000 + false + ''' + + +@pytest.fixture +def file_metadata(): + return { + 'Content-Length': '9001', + 'Last-Modified': 'SomeTime', + 'Content-Type': 'binary/octet-stream', + 'Etag': '"fba9dede5f27731c9771645a39863328"', + 'X-AMZ-SERVER-SIDE-ENCRYPTION': 'AES256' + } + + +@pytest.fixture +def version_metadata(): + return b''' + + + bucket + my + + + 5 + false + + my-image.jpg + 3/L4kqtJl40Nr8X8gdRQBpUMLUo + true + 2009-10-12T17:50:30.000Z + "fba9dede5f27731c9771645a39863328" + 434234 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + + my-image.jpg + QUpfdndhfd8438MNFDN93jdnJFkdmqnh893 + false + 2009-10-10T17:50:30.000Z + "9b2cf535f27731c974343645a3985328" + 166434 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + + my-image.jpg + UIORUnfndfhnw89493jJFJ + false + 2009-10-11T12:50:30.000Z + "772cf535f27731c974343645a3985328" + 64 + STANDARD + + 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a + mtd@amazon.com + + + ''' + +def location_response(location): + return ( + '\n' + '' + '{}' + ).format(location) + +def list_objects_response(keys, truncated=False): + response = ''' + + bucket + + + 1000''' + + response += '' + str(truncated).lower() + '' + response += ''.join(map( + lambda x: '{}'.format(x), + keys + )) + + response += '' + + return response.encode('utf-8') + +def bulk_delete_body(keys): + payload = '' + payload += '' + payload += ''.join(map( + lambda x: '{}'.format(x), + keys + )) + payload += '' + payload = payload.encode('utf-8') + + md5 = base64.b64encode(hashlib.md5(payload).digest()) + headers = { + 'Content-Length': str(len(payload)), + 'Content-MD5': md5.decode('ascii'), + 'Content-Type': 'text/xml', + } + + return (payload, headers) + +def build_folder_params(path): + return {'prefix': path.path, 'delimiter': '/'} + + +class TestProviderConstruction: + + def test_https(self, auth, credentials, settings): + provider = MyMinIOProvider(auth, {'host': 'securehost', + 'access_key': 'a', + 'secret_key': 's'}, settings) + assert provider.connection.is_secure + assert provider.connection.host == 'securehost' + assert provider.connection.port == 443 + + provider = MyMinIOProvider(auth, {'host': 'securehost:443', + 'access_key': 'a', + 'secret_key': 's'}, settings) + assert provider.connection.is_secure + assert provider.connection.host == 'securehost' + assert provider.connection.port == 443 + + def test_http(self, auth, credentials, settings): + provider = MyMinIOProvider(auth, {'host': 'normalhost:80', + 'access_key': 'a', + 'secret_key': 's'}, settings) + assert not provider.connection.is_secure + assert provider.connection.host == 'normalhost' + assert provider.connection.port == 80 + + provider = MyMinIOProvider(auth, {'host': 'normalhost:8080', + 'access_key': 'a', + 'secret_key': 's'}, settings) + assert not provider.connection.is_secure + assert provider.connection.host == 'normalhost' + assert provider.connection.port == 8080 + + +class TestValidatePath: + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_validate_v1_path_file(self, provider, file_metadata, mock_time): + file_path = 'foobah' + + params = {'prefix': '/' + file_path + '/', 'delimiter': '/'} + good_metadata_url = provider.bucket.new_key('/' + file_path).generate_url(100, 'HEAD') + bad_metadata_url = provider.bucket.generate_url(100) + aiohttpretty.register_uri('HEAD', good_metadata_url, headers=file_metadata) + aiohttpretty.register_uri('GET', bad_metadata_url, params=params, status=404) + + try: + wb_path_v1 = await provider.validate_v1_path('/' + file_path) + except Exception as exc: + pytest.fail(str(exc)) + + with pytest.raises(exceptions.NotFoundError) as exc: + await provider.validate_v1_path('/' + file_path + '/') + + assert exc.value.code == client.NOT_FOUND + + wb_path_v0 = await provider.validate_path('/' + file_path) + + assert wb_path_v1 == wb_path_v0 + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_validate_v1_path_folder(self, provider, folder_metadata, mock_time): + folder_path = 'Photos' + + params = {'prefix': '/' + folder_path + '/', 'delimiter': '/'} + good_metadata_url = provider.bucket.generate_url(100) + bad_metadata_url = provider.bucket.new_key('/' + folder_path).generate_url(100, 'HEAD') + aiohttpretty.register_uri( + 'GET', good_metadata_url, params=params, + body=folder_metadata, headers={'Content-Type': 'application/xml'} + ) + aiohttpretty.register_uri('HEAD', bad_metadata_url, status=404) + + try: + wb_path_v1 = await provider.validate_v1_path('/' + folder_path + '/') + except Exception as exc: + pytest.fail(str(exc)) + + with pytest.raises(exceptions.NotFoundError) as exc: + await provider.validate_v1_path('/' + folder_path) + + assert exc.value.code == client.NOT_FOUND + + wb_path_v0 = await provider.validate_path('/' + folder_path + '/') + + assert wb_path_v1 == wb_path_v0 + + @pytest.mark.asyncio + async def test_normal_name(self, provider, mock_time): + path = await provider.validate_path('/this/is/a/path.txt') + assert path.name == 'path.txt' + assert path.parent.name == 'a' + assert path.is_file + assert not path.is_dir + assert not path.is_root + + @pytest.mark.asyncio + async def test_folder(self, provider, mock_time): + path = await provider.validate_path('/this/is/a/folder/') + assert path.name == 'folder' + assert path.parent.name == 'a' + assert not path.is_file + assert path.is_dir + assert not path.is_root + + @pytest.mark.asyncio + async def test_root(self, provider, mock_time): + path = await provider.validate_path('/this/is/a/folder/') + assert path.name == 'folder' + assert path.parent.name == 'a' + assert not path.is_file + assert path.is_dir + assert not path.is_root + + +class TestCRUD: + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_download(self, provider, mock_time): + path = WaterButlerPath('/muhtriangle') + url = provider.bucket.new_key(path.path).generate_url(100, response_headers={'response-content-disposition': 'attachment'}) + aiohttpretty.register_uri('GET', url[:url.index('?')], body=b'delicious', auto_length=True) + + result = await provider.download(path) + content = await result.read() + + assert content == b'delicious' + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_download_version(self, provider, mock_time): + path = WaterButlerPath('/muhtriangle') + url = provider.bucket.new_key(path.path).generate_url( + 100, + query_parameters={'versionId': 'someversion'}, + response_headers={'response-content-disposition': 'attachment'}, + ) + aiohttpretty.register_uri('GET', url[:url.index('?')], body=b'delicious', auto_length=True) + + result = await provider.download(path, version='someversion') + content = await result.read() + + assert content == b'delicious' + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_download_display_name(self, provider, mock_time): + path = WaterButlerPath('/muhtriangle') + url = provider.bucket.new_key(path.path).generate_url(100, response_headers={'response-content-disposition': "attachment; filename*=UTF-8''tuna"}) + aiohttpretty.register_uri('GET', url[:url.index('?')], body=b'delicious', auto_length=True) + + result = await provider.download(path, displayName='tuna') + content = await result.read() + + assert content == b'delicious' + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_download_not_found(self, provider, mock_time): + path = WaterButlerPath('/muhtriangle') + url = provider.bucket.new_key(path.path).generate_url(100, response_headers={'response-content-disposition': 'attachment'}) + aiohttpretty.register_uri('GET', url[:url.index('?')], status=404) + + with pytest.raises(exceptions.DownloadError): + await provider.download(path) + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_download_folder_400s(self, provider, mock_time): + with pytest.raises(exceptions.DownloadError) as e: + await provider.download(WaterButlerPath('/cool/folder/mom/')) + assert e.value.code == 400 + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_upload_update(self, provider, file_content, file_stream, file_metadata, mock_time): + path = WaterButlerPath('/foobah') + content_md5 = hashlib.md5(file_content).hexdigest() + url = provider.bucket.new_key(path.path).generate_url(100, 'PUT') + metadata_url = provider.bucket.new_key(path.path).generate_url(100, 'HEAD') + aiohttpretty.register_uri('HEAD', metadata_url, headers=file_metadata) + aiohttpretty.register_uri('PUT', url, status=201, headers={'Etag': '"{}"'.format(content_md5)}) + + metadata, created = await provider.upload(file_stream, path) + + assert metadata.kind == 'file' + assert not created + assert aiohttpretty.has_call(method='PUT', uri=url) + assert aiohttpretty.has_call(method='HEAD', uri=metadata_url) + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_delete(self, provider, mock_time): + path = WaterButlerPath('/some-file') + url = provider.bucket.new_key(path.path).generate_url(100, 'DELETE') + aiohttpretty.register_uri('DELETE', url, status=200) + + await provider.delete(path) + + assert aiohttpretty.has_call(method='DELETE', uri=url) + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_folder_delete(self, provider, contents_and_self, mock_time): + path = WaterButlerPath('/some-folder/') + + params = {'prefix': 'some-folder/'} + query_url = provider.bucket.generate_url(100, 'GET') + aiohttpretty.register_uri( + 'GET', + query_url, + params=params, + body=contents_and_self, + status=200, + ) + + target_items = ['thisfolder/', 'thisfolder/item1', 'thisfolder/item2'] + delete_urls = [] + for i in target_items: + delete_url = provider.bucket.new_key(i).generate_url( + 100, + 'DELETE', + ) + delete_urls.append(delete_url) + aiohttpretty.register_uri('DELETE', delete_url, status=204) + + await provider.delete(path) + + assert aiohttpretty.has_call(method='GET', uri=query_url, params=params) + for delete_url in delete_urls: + assert aiohttpretty.has_call(method='DELETE', uri=delete_url) + + +class TestMetadata: + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_metadata_folder(self, provider, folder_metadata, mock_time): + path = WaterButlerPath('/darp/') + url = provider.bucket.generate_url(100) + params = build_folder_params(path) + aiohttpretty.register_uri('GET', url, params=params, body=folder_metadata, + headers={'Content-Type': 'application/xml'}) + + result = await provider.metadata(path) + + assert isinstance(result, list) + assert len(result) == 3 + assert result[0].name == ' photos' + assert result[1].name == 'my-image.jpg' + assert result[2].extra['md5'] == '1b2cf535f27731c974343645a3985328' + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_metadata_folder_self_listing(self, provider, contents_and_self, mock_time): + path = WaterButlerPath('/thisfolder/') + url = provider.bucket.generate_url(100) + params = build_folder_params(path) + aiohttpretty.register_uri('GET', url, params=params, body=contents_and_self) + + result = await provider.metadata(path) + + assert isinstance(result, list) + assert len(result) == 2 + for fobj in result: + assert fobj.name != path.path + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_just_a_folder_metadata_folder(self, provider, just_a_folder_metadata, mock_time): + path = WaterButlerPath('/') + url = provider.bucket.generate_url(100) + params = build_folder_params(path) + aiohttpretty.register_uri('GET', url, params=params, body=just_a_folder_metadata, + headers={'Content-Type': 'application/xml'}) + + result = await provider.metadata(path) + + assert isinstance(result, list) + assert len(result) == 1 + assert result[0].kind == 'folder' + + # @pytest.mark.asyncio + # @pytest.mark.aiohttpretty + # async def test_must_have_slash(self, provider, just_a_folder_metadata, mock_time): + # with pytest.raises(exceptions.InvalidPathError): + # await provider.metadata('') + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_metadata_file(self, provider, file_metadata, mock_time): + path = WaterButlerPath('/Foo/Bar/my-image.jpg') + url = provider.bucket.new_key(path.path).generate_url(100, 'HEAD') + aiohttpretty.register_uri('HEAD', url, headers=file_metadata) + + result = await provider.metadata(path) + + assert isinstance(result, metadata.BaseFileMetadata) + assert result.path == str(path) + assert result.name == 'my-image.jpg' + assert result.extra['md5'] == 'fba9dede5f27731c9771645a39863328' + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_metadata_file_missing(self, provider, mock_time): + path = WaterButlerPath('/notfound.txt') + url = provider.bucket.new_key(path.path).generate_url(100, 'HEAD') + aiohttpretty.register_uri('HEAD', url, status=404) + + with pytest.raises(exceptions.MetadataError): + await provider.metadata(path) + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_upload(self, provider, file_content, file_stream, file_metadata, mock_time): + path = WaterButlerPath('/foobah') + content_md5 = hashlib.md5(file_content).hexdigest() + url = provider.bucket.new_key(path.path).generate_url(100, 'PUT') + metadata_url = provider.bucket.new_key(path.path).generate_url(100, 'HEAD') + aiohttpretty.register_uri( + 'HEAD', + metadata_url, + responses=[ + {'status': 404}, + {'headers': file_metadata}, + ], + ) + aiohttpretty.register_uri('PUT', url, status=200, headers={'Etag': '"{}"'.format(content_md5)}), + + metadata, created = await provider.upload(file_stream, path) + + assert metadata.kind == 'file' + assert created + assert aiohttpretty.has_call(method='PUT', uri=url) + assert aiohttpretty.has_call(method='HEAD', uri=metadata_url) + + +class TestCreateFolder: + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_raise_409(self, provider, just_a_folder_metadata, mock_time): + path = WaterButlerPath('/alreadyexists/') + url = provider.bucket.generate_url(100, 'GET') + params = build_folder_params(path) + aiohttpretty.register_uri('GET', url, params=params, body=just_a_folder_metadata, + headers={'Content-Type': 'application/xml'}) + + with pytest.raises(exceptions.FolderNamingConflict) as e: + await provider.create_folder(path) + + assert e.value.code == 409 + assert e.value.message == 'Cannot create folder "alreadyexists", because a file or folder already exists with that name' + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_must_start_with_slash(self, provider, mock_time): + path = WaterButlerPath('/alreadyexists') + + with pytest.raises(exceptions.CreateFolderError) as e: + await provider.create_folder(path) + + assert e.value.code == 400 + assert e.value.message == 'Path must be a directory' + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_errors_out(self, provider, mock_time): + path = WaterButlerPath('/alreadyexists/') + url = provider.bucket.generate_url(100, 'GET') + params = build_folder_params(path) + create_url = provider.bucket.new_key(path.path).generate_url(100, 'PUT') + + aiohttpretty.register_uri('GET', url, params=params, status=404) + aiohttpretty.register_uri('PUT', create_url, status=403) + + with pytest.raises(exceptions.CreateFolderError) as e: + await provider.create_folder(path) + + assert e.value.code == 403 + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_errors_out_metadata(self, provider, mock_time): + path = WaterButlerPath('/alreadyexists/') + url = provider.bucket.generate_url(100, 'GET') + params = build_folder_params(path) + + aiohttpretty.register_uri('GET', url, params=params, status=403) + + with pytest.raises(exceptions.MetadataError) as e: + await provider.create_folder(path) + + assert e.value.code == 403 + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_creates(self, provider, mock_time): + path = WaterButlerPath('/doesntalreadyexists/') + url = provider.bucket.generate_url(100, 'GET') + params = build_folder_params(path) + create_url = provider.bucket.new_key(path.path).generate_url(100, 'PUT') + + aiohttpretty.register_uri('GET', url, params=params, status=404) + aiohttpretty.register_uri('PUT', create_url, status=200) + + resp = await provider.create_folder(path) + + assert resp.kind == 'folder' + assert resp.name == 'doesntalreadyexists' + assert resp.path == '/doesntalreadyexists/' + + +class TestOperations: + + # @pytest.mark.asyncio + # @pytest.mark.aiohttpretty + # async def test_copy(self, provider, file_metadata, mock_time): + # dest_path = WaterButlerPath('/dest') + # source_path = WaterButlerPath('/source') + # headers = {'x-amz-copy-source': '/{}/{}'.format(provider.settings['bucket'], source_path.path)} + + # metadata_url = provider.bucket.new_key(dest_path.path).generate_url(100, 'HEAD') + # url = provider.bucket.new_key(dest_path.path).generate_url(100, 'PUT', headers=headers) + + # aiohttpretty.register_uri('PUT', url, status=200) + # aiohttpretty.register_uri('HEAD', metadata_url, headers=file_metadata) + + # resp = await provider.copy(provider, source_path, dest_path) + + # # TODO: matching url content for request + # assert resp['kind'] == 'file' + # assert aiohttpretty.has_call(method='HEAD', uri=metadata_url) + # assert aiohttpretty.has_call(method='PUT', uri=url, headers=headers) + + @pytest.mark.asyncio + @pytest.mark.aiohttpretty + async def test_version_metadata(self, provider, version_metadata, mock_time): + path = WaterButlerPath('/my-image.jpg') + url = provider.bucket.generate_url(100, 'GET', query_parameters={'versions': ''}) + params = build_folder_params(path) + aiohttpretty.register_uri('GET', url, params=params, status=200, body=version_metadata) + + data = await provider.revisions(path) + + assert isinstance(data, list) + assert len(data) == 3 + + for item in data: + assert hasattr(item, 'extra') + assert hasattr(item, 'version') + assert hasattr(item, 'version_identifier') + + assert aiohttpretty.has_call(method='GET', uri=url, params=params) + + async def test_equality(self, provider, mock_time): + assert provider.can_intra_copy(provider) + assert provider.can_intra_move(provider)