Support custom text formats and recursive (#496)

* Add custom text types and recursive

* Add custom text types and recursive

* Fix format

* Update qdrant, Add pdf to unstructured

* Use unstructed as the default text extractor if installed

* Add tests for unstructured

* Update tests env for unstructured

* Fix error if last message is a function call, issue #569

* Remove csv, md and tsv from UNSTRUCTURED_FORMATS

* Update docstring of docs_path

* Update test for get_files_from_dir

* Update docstring of custom_text_types

* Fix missing search_string in update_context

* Add custom_text_types to notebook example

.github/workflows/build.yml

@@ -42,10 +42,6 @@ jobs:
           python -c "import autogen"
           pip install -e. pytest mock
           pip uninstall -y openai
-      - name: Install unstructured if not windows
-        if: matrix.os != 'windows-2019'
-        run: |
-          pip install "unstructured[all-docs]"
       - name: Test with pytest
         if: matrix.python-version != '3.10'
         run: |

.github/workflows/contrib-tests.yml

@@ -34,10 +34,14 @@ jobs:
         run: |
           python -m pip install --upgrade pip wheel
           pip install pytest
-      - name: Install qdrant_client when python-version is 3.10
-        if: matrix.python-version == '3.10' || matrix.python-version == '3.8'
+      - name: Install qdrant_client when python-version is 3.8 and 3.10
+        if: matrix.python-version == '3.8' || matrix.python-version == '3.10'
         run: |
           pip install qdrant_client[fastembed]
+      - name: Install unstructured when python-version is 3.9 and 3.11 and not windows
+        if: (matrix.python-version == '3.9' || matrix.python-version == '3.11') && matrix.os != 'windows-2019'
+        run: |
+          pip install unstructured[all-docs]
       - name: Install packages and dependencies for RetrieveChat
         run: |
           pip install -e .[retrievechat]